Image gradation processing apparatus and recording

ABSTRACT

An image processing system includes a frequency decomposition unit adapted to decompose an image signal into a high frequency component and a low frequency component, a high frequency separation unit adapted to separate the high frequency component into an invalid component caused by noise and other valid component, a conversion characteristic calculation unit adapted to calculate a conversion characteristic on the basis of the low frequency component, a gradation processing unit adapted to perform a gradation processing on the low frequency component and the valid component on the basis of the calculated conversion characteristic, and a frequency synthesis unit adapted to generate an image signal on which a gradation conversion has been performed by synthesizing the low frequency component on which the gradation conversion has been performed, the valid component on which a gradation conversion has been performed and the invalid component.

CROSS REFERENCE TO RELATED APPLICATION

This application is a continuation application of PCT/JP2007/067222filed on Sep. 4, 2007 and claims benefit of Japanese Application No.2006-247169 filed in Japan on Sep. 12, 2006, the entire contents ofwhich are incorporated herein by this reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an image processing system arranged toperform a gradation conversion on an image signal and a recording mediumrecording an image processing program for performing the gradationconversion on the image signal.

2. Description of the Related Art

As a gradation processing to be performed on an image signal, aspace-invariant method of using a single gradation conversion curve forthe image signal and a space-variant method of using a plurality ofgradation conversion curves different for each local region areproposed.

For example, Japanese Patent No. 3465226 discloses a technology fordividing the image signal into a plurality of regions on the basis oftexture information, performing a gradation conversion processing bycalculating gradation conversion curve for each region on the basis of ahistogram, and performing a weighting interpolation on the basis of adistance between the respective regions. With the configuration, it ispossible to perform the space-variant gradation processing and maintainthe continuity between the regions, and it is possible to obtain thehigh quality image signals in which light-dark crush is prevented alsofor an image having a wide dynamic range.

Also, Japanese Unexamined Patent Application Publication No. 8-56316discloses a technology for separating the image signal into a highfrequency component and a low frequency component, performing a contrastemphasis processing on the low frequency component, and synthesizing thelow frequency component after the contrast emphasis processing with thehigh frequency component. By employing such a technology, an emphasis onthe noise of the high frequency component is prevented, and it ispossible to obtain the high quality image signals.

Furthermore, Japanese Unexamined Patent Application Publication No.2004-128985 discloses a technology for estimating a noise amount foreach block unit on the basis of a noise model and performing differentnoise reducing processings for each block unit. By employing such atechnology, it is possible to perform a space-variant noise reducingprocessing, and it is possible to obtain the high quality image signalsin which degradation of the edge component is little.

SUMMARY OF THE INVENTION

According to an aspect of the present invention, there is provided animage processing system arranged to perform a gradation conversion on animage signal, the image processing system including: separation meansadapted to separate the image signal into an invalid component caused bynoise and other valid component; conversion means adapted to perform thegradation conversion on the valid component; and synthesis means adaptedto synthesize an image signal on which the gradation conversion has beenperformed on the basis of the valid component on which the gradationconversion has been performed and the invalid component.

Also, according to an aspect of the present invention, there is provideda recording medium recording an image processing program for instructinga computer to perform a gradation conversion on an image signal, theimage processing program instructing the computer to execute: aseparation step of separating the image signal into an invalid componentcaused by noise and other valid component; a conversion step ofperforming the gradation conversion on the valid component; and asynthesis step of synthesizing an image signal on which the gradationconversion has been performed on the basis of the valid component onwhich the gradation conversion has been performed and the invalidcomponent.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of a configuration of an image processingsystem according to a first embodiment of the present invention;

FIG. 2 is a block diagram of a configuration example of a frequencydecomposition unit according to the first embodiment;

FIG. 3A is an explanatory diagram for describing a wavelet transform,illustrating an image signal in a real space according to the firstembodiment;

FIG. 3B is an explanatory diagram for describing the wavelet transform,illustrating the signal after the first wavelet transform has beenperformed according to the first embodiment;

FIG. 3C is an explanatory diagram for describing the wavelet transform,illustrating the signal after the second wavelet transform has beenperformed according to the first embodiment;

FIG. 4 is a block diagram of a configuration example of a conversioncharacteristic calculation unit according to the first embodiment;

FIG. 5 is a block diagram of a configuration example of a high frequencyseparation unit according to the first embodiment;

FIG. 6 is a block diagram of a configuration example of a gradationprocessing unit according to the first embodiment;

FIG. 7 is an explanatory diagram for describing a division into regionsof a low frequency component in a synthesis operation for gradationconversion curves according to the first embodiment;

FIG. 8 is an explanatory diagram for describing distances d₁ to d₄between a target pixel and neighboring four regions in the synthesisoperation for gradation conversion curves according to the firstembodiment;

FIG. 9 is a block diagram of a configuration example of a frequencysynthesis unit according to the first embodiment;

FIG. 10 is a diagram illustrating another configuration example of theimage processing system according to the first embodiment;

FIG. 11 is a flow chart showing a main routine of an image processingprogram according to the first embodiment;

FIG. 12 is a flow chart showing a processing for a conversioncharacteristic calculation in step S3 of FIG. 11 according to the firstembodiment;

FIG. 13 is a flow chart showing a processing for a high frequencyseparation in step S4 of FIG. 11 according to the first embodiment;

FIG. 14 is a flow chart showing a gradation processing in step S5 ofFIG. 11 according to the first embodiment;

FIG. 15 is a block diagram of a configuration of an image processingsystem according to a second embodiment of the present invention;

FIG. 16 is a diagram illustrating a configuration of a Bayer-typeprimary color filter according to the second embodiment;

FIG. 17 is a diagram illustrating a configuration of a color-differenceline-sequential type complementary color filter according to the secondembodiment;

FIG. 18 is a block diagram of a configuration example of a frequencydecomposition unit according to the second embodiment;

FIG. 19 is a block diagram of a configuration example of a conversioncharacteristic calculation unit according to the second embodiment 2;

FIG. 20 is a block diagram of a configuration example of a highfrequency separation unit according to the second embodiment;

FIG. 21 is a block diagram of a configuration example of a gradationprocessing unit according to the second embodiment;

FIG. 22 is a flow chart showing a main routine of an image processingprogram according to the second embodiment;

FIG. 23 is a flow chart showing a processing for a conversioncharacteristic calculation in step S51 of FIG. 22 according to thesecond embodiment;

FIG. 24 is a flow chart showing a processing for a high frequencyseparation in step S52 of FIG. 22 according to the second embodiment;

FIG. 25 is a flow chart showing a gradation processing in step S53 ofFIG. 22 according to the second embodiment;

FIG. 26 is a block diagram of a configuration of an image processingsystem according to a third embodiment of the present invention;

FIG. 27A is an explanatory diagram for describing a DCT (discrete cosinetransform), illustrating an image signal in a real space according tothe third embodiment;

FIG. 27B is an explanatory diagram for describing the DCT (discretecosine transform), illustrating a signal in a frequency space after theDCT transform according to the third embodiment;

FIG. 28 is a block diagram of a configuration example of a highfrequency separation unit according to the third embodiment;

FIG. 29 is a flow chart showing a main routine of an image processingprogram according to the third embodiment;

FIG. 30 is a flow chart showing a processing for a high frequencyseparation in step S80 of FIG. 29 according to the third embodiment;

FIG. 31 is a block diagram of a configuration of an image processingsystem according to a fourth embodiment of the present invention;

FIG. 32 is a block diagram of a configuration example of a noisereducing unit according to the fourth embodiment;

FIG. 33 is a block diagram of a configuration example of a gradationprocessing unit according to the fourth embodiment;

FIG. 34 is a flow chart showing a main routine of an image processingprogram according to the fourth embodiment;

FIG. 35 is a flow chart showing a processing for a noise reduction instep S100 of FIG. 34 according to the fourth embodiment; and

FIG. 36 is a flow chart showing a gradation processing in step S102 ofFIG. 34 according to the fourth embodiment.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Hereinafter, embodiments of the present invention will be described withreference to the drawings.

First Embodiment

FIG. 1 to FIG. 14 illustrate a first embodiment of the presentinvention, and FIG. 1 is a block diagram of a configuration of an imageprocessing system.

The image processing system illustrated in FIG. 1 is an exampleconstituted as an image pickup system including an image pickup unit.

That is, the image processing system includes a lens system 100, anaperture 101, a CCD 102, an amplification unit 103, an A/D conversionunit (in the drawing, which is simply referred to as “A/D”) 104, abuffer 105, an exposure control unit 106, a focus control unit 107, anAF motor 108, a frequency decomposition unit 109 constituting separationmeans and frequency decomposition means, a buffer 110, a conversioncharacteristic calculation unit 111 constituting conversion means andconversion characteristic calculation means, a high frequency separationunit 112 constituting separation means and high frequency separationmeans, a gradation processing unit 113 constituting conversion means andgradation processing means, a buffer 114, a frequency synthesis unit 115constituting synthesis means and frequency synthesis means, a signalprocessing unit 116, an output unit 117, a control unit 118 constitutingcontrol means and doubling as noise estimation means and collectionmeans, an external I/F unit 119, and a temperature sensor 120.

An analog image signal captured and output via the lens system 100, theaperture 101, the CCD 102 is amplified by the amplification unit 103 andconverted into a digital signal by the A/D conversion unit 104.

The image signal from the A/D conversion unit 104 is transferred via thebuffer 105 to the frequency decomposition unit 109. The buffer 105 isconnected to the exposure control unit 106 and also to the focus controlunit 107.

The exposure control unit 106 is connected to the aperture 101, the CCD102, and the amplification unit 103. Also, the focus control unit 107 isconnected to the AF motor 108.

The signal from the frequency decomposition unit 109 is connected to thebuffer 110. The buffer 110 is connected to the conversion characteristiccalculation unit 111, the high frequency separation unit 112, and thegradation processing unit 113.

The conversion characteristic calculation unit 111 is connected to thegradation processing unit 113. The high frequency separation unit 112 isconnected to the gradation processing unit 113 and the buffer 114. Thegradation processing unit 113 is connected to the buffer 114.

The buffer 114 is connected via the frequency synthesis unit 115 and thesignal processing unit 116 to the output unit 117 such as a memory card.

The control unit 118 is composed, for example, of a micro computer. Thecontrol unit 118 is bi-directionally connected to the amplification unit103, the A/D conversion unit 104, the exposure control unit 106, thefocus control unit 107, the frequency decomposition unit 109, theconversion characteristic calculation unit 111, the high frequencyseparation unit 112, the gradation processing unit 113, the frequencysynthesis unit 115, the signal processing unit 116, and the output unit117, and is configured to control these units.

In addition, the external I/F unit 119 is also bi-directionallyconnected to the control unit 118. The external I/F unit 119 is aninterface provided with a power supply switch, a shutter button, a modebutton for performing switching of various modes for each shootingoperation, and the like.

Furthermore, the signal from the temperature sensor 120 is alsoconnected to the control unit 118. The temperature sensor 120 isarranged in a neighborhood of the CCD 102, and is configured tosubstantially measure the temperature of the CCD 102.

Next, the action of the image processing system illustrated in FIG. 1will be described along the flow of the image signal.

Before performing the shooting operation, the user sets image pickupconditions such as an ISO sensitivity via the external I/F unit 119.

After that, when the user performs a half press of the shutter buttonwhich is composed of a two-stage switch of the external I/F unit 119,the image processing system is turned into a pre-image pickup device.

The lens system 100 forms an optical image of a subject on an imagepickup plane of the CCD 102.

The aperture 101 regulates a passage range of the subject luminous fluxwhich has been formed into image by the lens system to change theluminance of the optical image formed on the image pickup plane of theCCD 102.

The CCD 102 photoelectrically converts the formed optical image andoutputs as an analog image signal. It should be noted that according tothe present embodiment, as the CCD 102, a monochrome single CCD isconsidered. But, the image pickup device is not limited to the CCD, butof course a CMOS or other image pickup devices may be used.

The analog signal output in this manner from the CCD 102 is amplified bythe amplification unit 103 by a predetermined amount while taking intoaccount the ISO sensitivity. Thereafter, the analog signal is convertedinto the digital signal by the A/D conversion unit 104 to be transferredto the buffer 105. It should be noted that according to the presentembodiment, the gradation width of the digitalized image signal is set,for example, as 12-bits.

The image signal stored in the buffer 105 is transferred to the exposurecontrol unit 106 and the focus control unit 107.

While taking into account the set ISO sensitivity, the shutter speed ata limit of image stability, and the like, the exposure control unit 106performs a control on an aperture value of the aperture 101, anelectronic shutter speed of the CCD 102, a gain of the amplificationunit 103, and the like to achieve the correct exposure on the basis ofthe image signal.

Also, the focus control unit 107 obtains a focus signal by detecting theedge intensity and controls the AF motor 108 so that the edge intensitybecomes the largest on the basis of the image signal.

In this way, after the focus adjustment, the exposure adjustment, or thelike is performed, when the user performs a full press of a shutterbutton which is composed of a two-stage switch of the external I/F unit119, the image processing system functions as a real shooting device.

After that, similarly to the pre shooting, the image signal istransferred to the buffer 105. The real shooting operation is performedon the basis of the exposure conditions calculated by the exposurecontrol unit 106 and the focus conditions calculated by the focuscontrol unit 107, and these conditions for each shooting operation aretransferred to the control unit 118.

The image signal in the buffer 105 obtained by the real shootingoperation is transferred to the frequency decomposition unit 109.

On the basis of the control of the control unit 118, the frequencydecomposition unit 109 performs a predetermined frequency decompositionon the transferred image signal to obtain a high frequency component anda low frequency component. Then, the frequency decomposition unit 109sequentially transfers the thus obtained high frequency component andthe low frequency component to the buffer 110. It should be noted thataccording to the present embodiment, for the frequency decomposition, itis supposed to employ the wavelet transform by two times.

The conversion characteristic calculation unit 111 reads the lowfrequency component from the buffer 110 to calculate gradationcharacteristics used for the gradation conversion processing on thebasis of the control of the control unit 118. It should be noted thataccording to the present embodiment, as the gradation conversionprocessing, a space-variant processing which uses a plurality ofgradation characteristics different for each local region is supposed.Then, the conversion characteristic calculation unit 111 transfers thecalculated gradation characteristics to the gradation processing unit113.

The high frequency separation unit 112 reads the high frequencycomponent from the buffer 110 to separate the high frequency componentinto an invalid component caused by noise and other valid component.Then, the high frequency separation unit 112 transfers the thusseparated valid component to the gradation processing unit 113 and theabove-mentioned invalid component to the buffer 114, respectively.

The gradation processing unit 113 reads the low frequency component fromthe buffer 110, the valid component in the high frequency component fromthe high frequency separation unit 112, and the gradation characteristicfrom the conversion characteristic calculation unit 111, respectively,on the basis of the control of the control unit 118. Then, the gradationprocessing unit 113 performs the gradation processing on the lowfrequency component and the valid component in the high frequencycomponent on the basis of the above-mentioned gradation characteristic.The gradation processing unit 113 transfers the low frequency componenton which the gradation processing has been performed and the validcomponent in the high frequency component on which the gradationprocessing has been performed to the buffer 114.

The frequency synthesis unit 115 reads the low frequency component onwhich the gradation processing has been performed, the valid componentin the high frequency component on which the gradation processing hasbeen performed, and the invalid component in the high frequencycomponent from the buffer 114, and synthesizes the image signal on whichthe gradation processing has been performed on the basis of thesecomponents under the control of the control unit 118. It should be notedthat according to the present embodiment, it is supposed to use theinverse wavelet transform as the frequency synthesis. Then, thefrequency synthesis unit 115 transfers the synthesized image signal tothe signal processing unit 116.

The signal processing unit 116 performs a known compression processingor the like on the image signal from the image signal the frequencysynthesis unit 115 and transfers the signal after the processing to theoutput unit 117 on the basis of the control of the control unit 118.

The output unit 117 records and saves the image signal output from thesignal processing unit 116 in the recording medium such as a memorycard.

Next, FIG. 2 is a block diagram of a configuration example of thefrequency decomposition unit 109.

The frequency decomposition unit 109 includes a data reading unit 200, abuffer 201, a horizontal high-pass filter (in the drawing, which issimply referred to as “horizontal high-pass”, and the same applies inthe following description) 202, a horizontal low-pass filter (in thedrawing, which is simply referred to as “horizontal low-pass”, and thesame applies in the following description) 203, a sub sampler 204, a subsampler 205, a vertical high-pass filter (in the drawing, which issimply referred to as “vertical high-pass”, and the same applies in thefollowing description) 206, a vertical low-pass filter (in the drawing,which is simply referred to as “vertical low-pass”, and the same appliesin the following description) 207, a vertical high-pass filter 208, avertical low-pass filter 209, a sub sampler 210, a sub sampler 211, asub sampler 212, a sub sampler 213, a switching unit 214, a datatransfer control unit 215, a basis function ROM 216, and a filtercoefficient reading unit 217.

The buffer 105 is connected via the data reading unit 200 to the buffer201.

The buffer 201 is connected to the horizontal high-pass filter 202 andthe horizontal low-pass filter 203.

The horizontal high-pass filter 202 is connected via the sub sampler 204to the vertical high-pass filter 206 and the vertical low-pass filter207. The horizontal low-pass filter 203 is connected via the sub sampler205 to the vertical high-pass filter 208 and the vertical low-passfilter 209.

The vertical high-pass filter 206 is connected to the sub sampler 210,the vertical low-pass filter 207 is connected to the sub sampler 211,the vertical high-pass filter 208 is connected to the sub sampler 212,and the vertical low-pass filter 209 is connected to the sub sampler213, respectively.

The sub sampler 210, the sub sampler 211, and the sub sampler 212 areconnected to the switching unit 214.

The sub sampler 213 is connected to the switching unit 214 and the datatransfer control unit 215. The switching unit 214 is connected to thebuffer 110. The data transfer control unit 215 is connected to thebuffer 201.

The basis function ROM 216 is connected to the filter coefficientreading unit 217. The filter coefficient reading unit 217 is connectedto the horizontal high-pass filter 202, the horizontal low-pass filter203, the vertical high-pass filter 206, the vertical low-pass filter207, the vertical high-pass filter 208, and the vertical low-pass filter209.

The control unit 118 is bi-directionally connected to the data readingunit 200, the switching unit 214, the data transfer control unit 215,and the filter coefficient reading unit 217 to control these units.

The basis function ROM 216 records filter coefficients used for thewavelet transform such as Harr function or Daubechies function. Amongthese, for example, the coefficient of the high-pass filter in the Harrfunction is represented by Numeric Expression 1 and the coefficient ofthe low-pass filter is represented by Numeric Expression 2,respectively.High-pass filter coefficient={0.5,−0.5}  [Expression 1]Low-pass filter coefficient={0.5,0.5}  [Expression 2]

It should be noted that these filter coefficients are commonly used inthe horizontal direction and the vertical direction.

The filter coefficient reading unit 217 reads the filter coefficientsfrom the basis function ROM 216, transfers the high-pass filtercoefficient to the horizontal high-pass filter 202, the verticalhigh-pass filter 206, and the vertical high-pass filter 208, andtransfers the low-pass filter coefficient to the horizontal low-passfilter 203, the vertical low-pass filter 207, and the vertical low-passfilter 209, respectively, on the basis of the control of the controlunit 118.

In this way, after the filter coefficients are transferred to therespective high-pass filters and the respective low-pass filters, on thebasis of the control of the control unit 118, the data reading unit 200reads the image signal from the buffer 105 to be transferred to thebuffer 201. It should be noted that in the following description, theimage signal read from the buffer 105 and stored on the buffer 201 isset as L₀.

The image signal on the buffer 201 is subjected to the filteringprocessing in the horizontal direction and the vertical direction by thehorizontal high-pass filter 202, the horizontal low-pass filter 203, thevertical high-pass filter 206, the vertical low-pass filter 207, thevertical high-pass filter 208, and the vertical low-pass filter 209.

At this time, the sub sampler 204 and the sub sampler 205 perform thesub sampling on the input image signal in the horizontal direction into½, and the sub sampler 210, the sub sampler 211, the sub sampler 212,and the sub sampler 213 performs the sub sampling on the input imagesignal in the vertical direction into ½.

Therefore, the output of the sub sampler 210 provides a high frequencycomponent Hs1 _(ij) in the slanted direction in the transform performedfor the first time, the output of the sub sampler 211 provides afirst-order high frequency component Hh1 _(ij) in the horizontaldirection in the transform performed for the first time, the output ofthe sub sampler 212 provides a first-order high frequency component Hv1_(ij) in the vertical direction in the transform performed for the firsttime, the output of the sub sampler 213 provides a first-order lowfrequency component L1 _(ij) in the transform performed for the firsttime, respectively. Herein, suffixes i and j mean coordinates in x and ydirections in the first-order signal after the transform.

FIGS. 3A to 3C are explanatory diagrams for describing the wavelettransform: FIG. 3A illustrates the image signal in the real space, FIG.3B illustrates the signal after the wavelet transform is performed forthe first time, and FIG. 3C illustrates the signal after the wavelettransform is performed for the second time, respectively.

When the wavelet transform is performed for the first time on the imagesignal in the real space as illustrated in FIG. 3A, the signal becomesas illustrated in FIG. 3B. Also, FIG. 3B illustrates the first-orderhigh frequency component Hs1 ₀₀ in the slanted direction, thefirst-order high frequency component Hh1 ₀₀ in the horizontal direction,and the first-order high frequency component Hv1 ₀₀ in the verticaldirection corresponding to the low frequency component L1 ₀₀.

In the transform performed for the first time, the three first-orderhigh frequency components Hs1 _(ij), Hh1 _(ij), and Hv1 _(ij)corresponding to the first-order low frequency component L1 _(ij) of onepixel, are all one pixel.

On the basis of the control of the control unit 118, the switching unit214 sequentially transfers the above-mentioned three first-order highfrequency components Hs1 _(ij), Hh1 _(ij), and Hv1 _(ij) and thefirst-order low frequency component L1 _(ij) to the buffer 110.

Also, the data transfer control unit 215 transfers the first-order lowfrequency component L1 _(ij) from the sub sampler 213 to the buffer 201on the basis of the control of the control unit 118.

As the filtering processing similar to the above is performed on thefirst-order low frequency component L1 _(ij) on the buffer 201, threesecond-order high frequency components Hs2 _(kl), Hh2 _(kl), and Hv2_(kl) and a second-order low frequency component L2 _(kl) are output.Herein, suffixes k and l mean coordinates in the x and y directions inthe second-order signal after the transform.

FIG. 3C illustrates the signal in such a transform performed for thesecond time.

As illustrated in FIG. 3C, in the transform performed for the secondtime, the second-order high frequency component in the slanted directioncorresponding to the second-order low frequency component L2 ₀₀ of onepixel becomes Hs2 ₀₀, the second-order high frequency component in thehorizontal direction becomes Hh2 ₀₀, and the second-order high frequencycomponent in the vertical direction becomes Hv2 ₀₀, all of which are onepixel, but the first-order high frequency components in thecorresponding slanted direction become Hs1 ₀₀, Hs1 ₁₀, Hs1 ₀₁, and Hs1₁₁, the first-order high frequency components in the horizontaldirection become Hh1 ₀₀, Hh1 ₁₀, Hh1 ₀₁, and Hh1 ₁₁, and the first-orderhigh frequency components in the vertical direction become Hv1 ₀₀, Hv1₁₀, Hv1 ₀₁, and Hv1 ₁₁, all of which are four pixels. Theabove-mentioned procedure is repeatedly performed until decomposition ata predetermined n (n is an integer equal to or larger than 1, andaccording to the present embodiment, as described above, n=2 issupposed) stage is performed on the basis of the control of the controlunit 118.

Next, FIG. 4 is a block diagram of a configuration example of theconversion characteristic calculation unit 111.

The conversion characteristic calculation unit 111 includes a divisionunit 300 constituting division means, a buffer 301, a correct rangeextraction unit 302 constituting correct range extraction means, an edgecalculation unit 303 constituting region-of-interest setting means andedge calculation means, a histogram creation unit 304 constitutinghistogram creation means, a gradation conversion curve calculation unit305 constituting gradation conversion curve calculation means, and abuffer 306.

The buffer 110 is connected via the division unit 300 to the buffer 301.

The buffer 301 is connected to the correct range extraction unit 302 andthe histogram creation unit 304. The correct range extraction unit 302is connected via the edge calculation unit 303 to the histogram creationunit 304.

The histogram creation unit 304 is connected via the gradationconversion curve calculation unit 305 and the buffer 306 to thegradation processing unit 113.

The control unit 118 is bi-directionally connected to the division unit300, the correct range extraction unit 302, the edge calculation unit303, the histogram creation unit 304, and the gradation conversion curvecalculation unit 305 to control these units.

Subsequently, a description will be given of the action of theconversion characteristic calculation unit 111.

The division unit 300 reads the low frequency component of the imagesignal from the buffer 110 on the basis of the control of the controlunit 118 and divides the low frequency component into regions of apredetermined size shown in FIG. 7, for example, a 32×32 pixel size, sothat the respective regions are not overlapped one another. Herein, FIG.7 is an explanatory diagram for describing the division into the regionsof the low frequency component in the synthesis operation of thegradation conversion curves. Then, the division unit 300 sequentiallytransfers the divided regions to the buffer 301.

The correct range extraction unit 302 reads the low frequency componentsfrom the buffer 301 for each local region unit on the basis of thecontrol of the control unit 118. The correct range extraction unit 302compares the low frequency components with the pre-set threshold relatedto the dark part (by way of an example, in the case of 12-bit gradation,for example, 128) and the pre-set threshold related to the light part(in the case of the 12-bit gradation, for example, 3968), and transfersthe low frequency components which are equal to or larger than thethreshold of the dark part and also equal to or smaller than thethreshold of the light part as the correct exposure range to the edgecalculation unit 303.

The edge calculation unit 303 reads the low frequency components in thecorrect exposure range from the correct range extraction unit 302 on thebasis of the control of the control unit 118, and uses a Laplacianfilter or the like to calculate the known edge intensity. The edgecalculation unit 303 transfers the calculated edge intensity to thehistogram creation unit 304.

The histogram creation unit 304 selects a pixel having an edge intensitywhich is equal to or larger than the pre-set threshold (in the case ofthe above-mentioned 12-bit gradation, for example, 64) regarding theedge intensity from the edge calculation unit 303, and reads the lowfrequency components at the corresponding pixel positions from thebuffer 301 on the basis of the control of the control unit 118. Then,the histogram creation unit 304 creates a histogram related to the readlow frequency components and transfers the created histogram to thegradation conversion curve calculation unit 305.

The gradation conversion curve calculation unit 305 accumulates andfurthermore normalizes the histograms from the histogram creation unit304 on the basis of the control of the control unit 118 to calculate thegradation conversion curve. The normalization is performed whilefollowing the gradation of the image signal. In the case of theabove-mentioned 12-bit gradation, the normalization is performed so asto have the range of 0 to 4095. The gradation conversion curvecalculation unit 305 transfers the calculated gradation conversion curveto the buffer 306.

It should be noted that the respective processings in the correct rangeextraction unit 302, the edge calculation unit 303, the histogramcreation unit 304, and the gradation conversion curve calculation unit305 are performed in synchronization for each local region unit on thebasis of the control of the control unit 118.

Next, FIG. 5 is a block diagram of a configuration example of the highfrequency separation unit 112.

The high frequency separation unit 112 includes a low frequencycomponent extraction unit 400, a gain calculation unit 401 constitutingnoise estimation means and collection means, a standard value assigningunit 402 constituting noise estimation means and assigning means, aparameter ROM 403 constituting noise estimation means and recordingmeans, a parameter selection unit 404 constituting noise estimationmeans and parameter selection means, an interpolation unit 405constituting noise estimation means and interpolation means, a highfrequency component extraction unit 406, an average calculation unit 407constituting setting means and average calculation means, an upper limitand lower limit setting unit 408 constituting setting means and upperlimit and lower limit setting means, and a determination unit 409constituting determination means.

The buffer 110 is connected to the low frequency component extractionunit 400 and the high frequency component extraction unit 406. The lowfrequency component extraction unit 400 is connected to the parameterselection unit 404.

The gain calculation unit 401, the standard value assigning unit 402,and the parameter ROM 403 are connected to the parameter selection unit404. The parameter selection unit 404 is connected via the interpolationunit 405 to the upper limit and lower limit setting unit 408.

The high frequency component extraction unit 406 is connected to theaverage calculation unit 407 and the determination unit 409. The averagecalculation unit 407 is connected via the upper limit and lower limitsetting unit 408 to the determination unit 409.

The determination unit 409 is connected to the gradation processing unit113 and the buffer 114.

The control unit 118 is bi-directionally connected to the low frequencycomponent extraction unit 400, the gain calculation unit 401, thestandard value assigning unit 402, the parameter selection unit 404, theinterpolation unit 405, the high frequency component extraction unit406, the average calculation unit 407, the upper limit and lower limitsetting unit 408, and the determination unit 409 to control these units.

Subsequently, a description will be given of the action of the highfrequency separation unit 112.

The low frequency component extraction unit 400 sequentially extractsthe low frequency components from the buffer 110 for each pixel on thebasis of the control of the control unit 118. It should be noted thataccording to the present embodiment, it is supposed to perform thewavelet transform by two times. In this case, the low frequencycomponent extracted from the buffer 110 by the low frequency componentextraction unit 400 becomes the second-order low frequency component L2_(kl) as illustrated in FIG. 3C.

On the basis of the information related to the ISO sensitivity and theexposure condition transferred from the control unit 118, the gaincalculation unit 401 calculates the gain information in theamplification unit 103 and transfers the calculated gain information tothe parameter selection unit 404.

Also, the control unit 118 obtains temperature information of the CCD102 from the temperature sensor 120 and transfers the thus obtainedtemperature information to the parameter selection unit 404.

On the basis of the control of the control unit 118, in a case where atleast one of the above-mentioned gain information and the temperatureinformation cannot be obtained, the standard value assigning unit 402transfers a standard value of the information that cannot be obtained tothe parameter selection unit 404.

The parameter selection unit 404 searches the parameter ROM 403 for aparameter of a reference noise model used for estimating the noiseamount on the basis of the pixel value of the target pixel from the lowfrequency component extraction unit 400, the gain information from thegain calculation unit 401 or the standard value assigning unit 402, andthe temperature information from the control unit 118 or the standardvalue assigning unit 402. Then, the parameter selection unit 404transfers the searched parameter to the interpolation unit 405. Also,the parameter selection unit 404 transfers the image signal of the lowfrequency component from the low frequency component extraction unit 400to the interpolation unit 405.

The interpolation unit 405 calculates a noise amount N related to thelow frequency component on the basis of the parameter of the referencenoise model and transfers the calculated noise amount N to the upperlimit and lower limit setting unit 408.

It should be noted that to be more specific, the above-mentionedcalculation of the noise amount N based on the parameter ROM 403, theparameter selection unit 404, and the interpolation unit 405 can berealized through the technology disclosed in Japanese Unexamined PatentApplication Publication No. 2004-128985 described above, for example.

The high frequency component extraction unit 406 extracts the highfrequency component corresponding to the low frequency componentextracted by the low frequency component extraction unit 400 and thehigh frequency components located in the neighborhood of the highfrequency component on the basis of the control of the control unit 118.

For example, in a case where the second-order low frequency component L2₀₀ illustrated in FIG. 3C is extracted as the low frequency component,the high frequency components corresponding to the second-order lowfrequency component L2 ₀₀ become total three pixels of Hs2 ₀₀, Hh2 ₀₀,and Hv2 ₀₀ which are the second-order high frequency components andtotal 12 pixels of Hs1 ₀₀, Hs1 ₁₀, Hs1 ₀₁, Hs1 ₁₁, Hh1 ₀₀, Hh1 ₁₀, Hh1₀₁, Hh1 ₁₁, Hv1 ₀₀, Hv1 ₁₀, Hv1 ₀₁, and Hv1 ₁₁ which are the first-orderhigh frequency components.

Also, as the high frequency component located in a neighborhood of thehigh frequency components, for example, a region of 2×2 pixels includingthe corresponding high frequency component is selected.

The high frequency component extraction unit 406 sequentially transfersthe high frequency component corresponding to the low frequencycomponent and the high frequency components located in the neighborhoodof the high frequency component to the average calculation unit 407, andsequentially transfers the high frequency component corresponding to thelow frequency component to the determination unit 409.

On the basis of the control of the control unit 118, from the highfrequency component corresponding to the low frequency component and thehigh frequency components located in the neighborhood of the highfrequency component, the average calculation unit 407 calculates anaverage value AV and transfers the calculated average value AV to theupper limit and lower limit setting unit 408.

On the basis of the control of the control unit 118, by using theaverage value AV from the average calculation unit 407 and the noiseamount N from the interpolation unit 405, the upper limit and lowerlimit setting unit 408 sets an upper limit App_Up and a lower limitApp_Low for distinguishing the valid component and the invalid componentas represented by Numeric Expression 3 as follows.App_Up=AV+N/2App_Low=AV−N/2  [Expression 3]

The upper limit and lower limit setting unit 408 transfers the thus setupper limit App_Up and the lower limit App_Low to the determination unit409.

On the basis of the control of the control unit 118, the determinationunit 409 reads the high frequency component corresponding to the lowfrequency component from the high frequency component extraction unit406 and also reads the upper limit App_Up and the lower limit App_Lowshown in Numeric Expression 3 from the upper limit and lower limitsetting unit 408. Then, in a case where the high frequency component isin range between the upper limit App_Up and the lower limit App_Low (forexample, in a range equal to or larger than the lower limit App_Low andalso equal to or smaller than the upper limit App_Up), the determinationunit 409 determines that the high frequency component is the invalidcomponent caused by the noise and transfers the high frequency componentto the buffer 114. On the other hand, in a case where the high frequencycomponent exceeds the upper limit App_Up (larger than the upper limitApp_Up) or falls short of the lower limit App_Low (smaller than thelower limit App_Low), the determination unit 409 determines that thehigh frequency component is the valid component and transfers the highfrequency component to the gradation processing unit 113.

It should be noted that the respective processings in the averagecalculation unit 407, the upper limit and lower limit setting unit 408,and the determination unit 409 described above are performed insynchronization for the respective pixels of the corresponding highfrequency components on the basis of the control of the control unit118.

Next, FIG. 6 is a block diagram of a configuration example of thegradation processing unit 113.

The gradation processing unit 113 is configured by including a lowfrequency component extraction unit 500 constituting first extractionmeans, a distance calculation unit 501 constituting distance calculationmeans, a gradation conversion equation setting unit 502 constitutinggradation conversion equation setting means, a buffer 503, a highfrequency component extraction unit 504 constituting second extractionmeans, and a gradation conversion unit 505 constituting gradationconversion means.

The conversion characteristic calculation unit 111 is connected to thegradation conversion equation setting unit 502.

The buffer 110 is connected to the low frequency component extractionunit 500. The low frequency component extraction unit 500 is connectedto the distance calculation unit 501 and the gradation conversion unit505. The distance calculation unit 501 is connected to the gradationconversion unit 505 via the gradation conversion equation setting unit502 and the buffer 503.

The high frequency separation unit 112 is connected via the highfrequency component extraction unit 504 to the gradation conversion unit505.

The gradation conversion unit 505 is connected to the buffer 114.

The control unit 118 is bi-directionally connected to the low frequencycomponent extraction unit 500, the distance calculation unit 501, thegradation conversion equation setting unit 502, the high frequencycomponent extraction unit 504, and the gradation conversion unit 505 tocontrol these units.

Subsequently, a description will be given of the action of the gradationprocessing unit 113.

The low frequency component extraction unit 500 sequentially extractsthe low frequency components from the buffer 110 for each pixel on thebasis of the control of the control unit 118. It should be noted thataccording to the present embodiment, as described above, it is supposedto perform the wavelet transform by two times. In this case, the targetpixel of the low frequency component extracted by the low frequencycomponent extraction unit 500 from the buffer 110 becomes thesecond-order low frequency component L2 _(kl) as illustrated in FIG. 3C.

The low frequency component extraction unit 500 transfers the extractedlow frequency components to the distance calculation unit 501 and thegradation conversion unit 505.

The distance calculation unit 501 calculates distances between thetarget pixel extracted by the low frequency component extraction unit500 and four regions in a neighborhood of the target pixel.

FIG. 8 is an explanatory diagram of the distances between the targetpixel and the neighboring four regions d₁ to d₄ in the synthesisoperation of the gradation conversion curves.

The distances between the target pixel and the neighboring four regionsare respectively calculated as a distance between the target pixel andthe center of the respective regions. In the following description, thecalculated distances between the target pixel and the neighboring fourregions are represented by d_(m) (m=1 to 4), and the respectivegradation conversion curves of the neighboring four regions arerepresented by T_(m)( ). The distance calculation unit 501 transfers thecalculated distances d_(m) to the gradation conversion equation settingunit 502.

On the basis of the control of the control unit 118, the gradationconversion equation setting unit 502 reads the distances d_(m) from thedistance calculation unit 501 and also reads the corresponding gradationconversion curve T_(m)( ) of the neighboring four regions from theconversion characteristic calculation unit 111 to set the gradationconversion equation with respect to the target pixel as shown in NumericExpression 4 as follows.

$\begin{matrix}{{P^{\prime} = {\frac{1}{D}\left( {\frac{T_{1}(P)}{d_{1}} + \frac{T_{2}(P)}{d_{2}} + \frac{T_{3}(P)}{d_{3}} + \frac{T_{4}(P)}{d_{4}}} \right)}}{{{Where}\mspace{14mu} D} = {\frac{1}{d_{1}} + \frac{1}{d_{2}} + \frac{1}{d_{3}} + \frac{1}{d_{4}}}}} & \left\lbrack {{Expression}\mspace{14mu} 4} \right\rbrack\end{matrix}$

Herein, P in Numeric Expression 4 means a pixel of a target of thegradation conversion processing, and P′ means a pixel after thegradation conversion processing, respectively.

The gradation conversion equation setting unit 502 transfers thegradation conversion equation set as shown in Numeric Expression 4 tothe buffer 503.

On the other hand, the high frequency component extraction unit 504extracts the extracted high frequency components corresponding to thelow frequency components extracted by the low frequency componentextraction unit 500 from the high frequency separation unit 112 on thebasis of the control of the control unit 118. According to the presentembodiment, as the target pixel of the low frequency component is thesecond-order low frequency component L2 _(kl) shown in FIG. 3C, theextracted high frequency components becomes total three pixels includingone pixel each from the second-order high frequency components Hs2_(kl), Hh2 _(kl), and Hv2 _(kl) and total 12 pixels including fourpixels each from the first-order high frequency components Hs1 ij, Hh1ij, and Hv1 ij. Then, the high frequency component extraction unit 504transfers the extracted high frequency components to the gradationconversion unit 505.

After that, in a case where the high frequency component from the highfrequency component extraction unit 504 exists, the gradation conversionunit 505 reads the high frequency component and also reads the gradationconversion equation shown in Numeric Expression 4 from the buffer 503.On the basis of the read gradation conversion equation, the gradationconversion unit 505 performs the gradation conversion on the highfrequency components. The gradation conversion unit 505 transfers thehigh frequency component after the gradation conversion to the buffer114. On the other hand, in a case where it is determined that thecorresponding high frequency component is the invalid component and theextracted high frequency component does not exist, on the basis of thecontrol of the control unit 118, the gradation conversion unit 505cancels the gradation conversion on the high frequency component.

Also, the gradation conversion unit 505 reads the low frequencycomponent from the low frequency component extraction unit 500 and thegradation conversion equation shown in Numeric Expression 4 from thebuffer 503, respectively, to perform the gradation conversion on the lowfrequency component. The gradation conversion unit 505 transfers the lowfrequency component after the gradation conversion to the buffer 114.

Next, FIG. 9 is a block diagram of a configuration example of thefrequency synthesis unit 115.

The frequency synthesis unit 115 is configured by including a datareading unit 600, a switching unit 601, an up sampler 602, an up sampler603, an up sampler 604, an up sampler 605, a vertical high-pass filter606, a vertical low-pass filter 607, a vertical high-pass filter 608, avertical low-pass filter 609, an up sampler 610, an up sampler 611, ahorizontal high-pass filter 612, a horizontal low-pass filter 613, abuffer 614, a data transfer control unit 615, a basis function ROM 616,and a filter coefficient reading unit 617.

The buffer 114 is connected via the data reading unit 600 to theswitching unit 601. The switching unit 601 is connected to the upsampler 602, the up sampler 603, the up sampler 604, and the up sampler605. The up sampler 602 is connected to the vertical high-pass filter606, the up sampler 603 is connected to the vertical low-pass filter607, the up sampler 604 is connected to the vertical high-pass filter608, and the up sampler 605 is connected to the vertical low-pass filter609.

The vertical high-pass filter 606 and the vertical low-pass filter 607are connected to the up sampler 610, and the vertical high-pass filter608 and the vertical low-pass filter 609 are connected to the up sampler611. The up sampler 610 is connected to the horizontal high-pass filter612, and the up sampler 611 is connected to the horizontal low-passfilter 613. The horizontal high-pass filter 612 and the horizontallow-pass filter 613 are connected to the buffer 614. The buffer 614 isconnected to the signal processing unit 116 and the data transfercontrol unit 615.

The data transfer control unit 615 is connected to the switching unit601.

The basis function ROM 616 is connected to the filter coefficientreading unit 617. The filter coefficient reading unit 617 is connectedto the vertical high-pass filter 606, the vertical low-pass filter 607,the vertical high-pass filter 608, the vertical low-pass filter 609, thehorizontal high-pass filter 612, and the horizontal low-pass filter 613.

The control unit 118 is bi-directionally connected to the data readingunit 600, the switching unit 601, the data transfer control unit 615,and the filter coefficient reading unit 617 to control these units.

Subsequently, a description will be given of the action of the frequencysynthesis unit 115.

The basis function ROM 616 records a filter coefficient used for theinverse wavelet transform such as the Harr function or the Daubechiesfunction.

On the basis of the control of the control unit 118, the filtercoefficient reading unit 617 reads filter coefficient from the basisfunction ROM 616. The filter coefficient reading unit 617 transfers thehigh-pass filter coefficient to the vertical high-pass filter 606, thevertical high-pass filter 608, and the horizontal high-pass filter 612and the low-pass filter coefficient to the vertical low-pass filter 607,the vertical low-pass filter 609, the horizontal low-pass filter 613,respectively.

After the filter coefficients are transferred, on the basis of thecontrol of the control unit 118, the data reading unit 600 reads the lowfrequency component on which the gradation processing has been performedand the valid component at the n-stage in the high frequency componenton which the gradation processing has been performed and the invalidcomponent at the n-stage in the high frequency component from the buffer114 to be transferred to the switching unit 601. It should be noted thatthe valid component at the n-stage in the high frequency component onwhich the gradation processing has been performed and the invalidcomponent at the n-stage in the high frequency component are theintegrated high frequency component at the n-stage when read by the datareading unit 600.

On the basis of the control of the control unit 118, the switching unit601 transfers the high frequency components in the slanted direction viathe up sampler 602 to the vertical high-pass filter 606, the highfrequency components in the horizontal direction via the up sampler 603to the vertical low-pass filter 607, the high frequency components inthe vertical direction via the up sampler 604 to the vertical high-passfilter 608, and the low frequency components via the up sampler 605 tothe vertical low-pass filter 609, respectively, to execute the filteringprocessing in the vertical direction.

The frequency components from the vertical high-pass filter 606 and thevertical low-pass filter 607 are transferred via the up sampler 610 tothe horizontal high-pass filter 612, and the frequency components fromthe vertical high-pass filter 608 and the vertical low-pass filter 609are transferred via the up sampler 611 to the horizontal low-pass filter613, and then the filtering processing in the horizontal direction isperformed.

The frequency components from the horizontal high-pass filter 612 andthe horizontal low-pass filter 613 are transferred to the buffer 614 tobe synthesized into one, thus generating the low frequency component atthe (n−1)-th stage.

At this time, the up sampler 602, the up sampler 603, the up sampler604, and the up sampler 605 performs up sampling of the input frequencycomponent double in the vertical direction, and the up sampler 610 andthe up sampler 611 performs up sampling of the input frequency componentdouble in the horizontal direction.

The data transfer control unit 615 transfers the low frequencycomponents to the switching unit 601 on the basis of the control of thecontrol unit 118.

On the basis of the control of the control unit 118, the data readingunit 600 reads from the three types of high frequency components in theslanted direction, the horizontal direction, and the vertical directionat the (n−1)-th stage from the buffer 114 to be transferred to theswitching unit 601. Then, as the filtering processing similar to theabove is performed on the frequency at the stage number of thedecomposition (n−1), the low frequency component at the (n−2)-th stageis output to the buffer 614.

The above-mentioned procedure is repeatedly performed until the controlunit 118 performs the synthesis at a predetermined n-th stage. With theconfiguration, in the end, the low frequency component at the zero-thstage is output to the buffer 614 and the low frequency component at thezero-th stage is transferred to the signal processing unit 116 as theimage signal on which the gradation conversion has been performed.

It should be noted that in the above, the image processing system inwhich the image pickup unit including the lens system 100, the aperture101, the CCD 102, the amplification unit 103, the A/D conversion unit104, the exposure control unit 106, the focus control unit 107, the AFmotor 108, and the temperature sensor 120 is integrated has beendescribed. However, the image processing system is not necessarilylimited to the above-mentioned configuration. For example, asillustrated in FIG. 10, the image pickup unit may be provided as aseparated body. That is, in the image processing system illustrated inFIG. 10, the separated image pickup unit performs the image pickup, andan image signal recorded on a recording medium such as a memory card inan unprocessed raw data state is read out from the recording medium tobe processed. It should be noted that at this time, associatedinformation related to the image signal like the temperature of theimage pickup device, the exposure conditions, and the like, for eachshooting operation is recorded on a header unit or the like. It shouldbe noted that transmission of various pieces of information from theseparated image pickup unit to the image processing system is notnecessarily performed via a recording medium, and may be performed via acommunication circuit or the like.

Herein, FIG. 10 is a diagram illustrating another configuration exampleof the image processing system.

The image processing system illustrated in FIG. 10 has a configurationin which with respect to the image processing system illustrated in FIG.1, the lens system 100, the aperture 101, the CCD 102, the amplificationunit 103, the A/D conversion unit 104, the exposure control unit 106,the focus control unit 107, the AF motor 108, and the temperature sensor120 are omitted, and an input unit 700 and an header informationanalysis unit 701 are added. Other basic configuration in the imageprocessing system illustrated in FIG. 10 is similar to that illustratedin FIG. 1. Therefore, the same components are allocated with the samenames and reference numerals to appropriately omit the descriptionthereof, and only a different part will be mainly described.

The input unit 700 is connected to the buffer 105 and the headerinformation analysis unit 701. The control unit 118 is bi-directionallyconnected to the input unit 700 and the header information analysis unit701 to control these units.

Next, a different action in the image processing system illustrated inFIG. 10 is as follows.

For example, when a reproduction operation is started via the externalI/F unit 119 such as a mouse or a key board, the image signal and theheader information saved on the recording medium such as a memory cardare read via the input unit 700.

Among the information read from the input unit 700, the image signal istransferred to the buffer 105, and the header information is transferredto the header information analysis unit 701, respectively.

The header information analysis unit 701 extracts the information foreach shooting operation (that is, the exposure conditions, thetemperature of the image pickup device, and the like, which aredescribed above) to be transferred to the control unit 118 on the basisof the header information transferred from the input unit 700.

The processing in the following stage is similar to that of the imageprocessing system illustrated in FIG. 1.

Furthermore, in the above, it is supposed to perform the processing byway of the hardware, but the configuration is not necessarily limited tothe above. For example, the image signal from the CCD 112 is recorded onthe recording medium such as a memory card as raw data without applyingthe process, and also the associated information such as image pickupconditions (for example, the temperature of the image pickup device, theexposure conditions, and the like, for each shooting operation from thecontrol unit 118) is recorded in the recording medium as the headerinformation. Then, the processing can be performed as the computer isallowed to execute the image processing program which is separatesoftware to instruct the computer to read the information of therecording medium. It should be noted that the transmission of variouspieces of information from the image pickup unit to the computer is notnecessarily performed via the recording medium and may be performed viaa communication line or the like.

FIG. 11 is a flow chart showing a main routine of an image processingprogram.

When the processing is started, first, the image signal is read, andalso the header information such as the temperature and the exposureconditions of the image pickup device is read (step S1).

Next, by performing the frequency decomposition such as the wavelettransform, the high frequency component and the low frequency componentare obtained (step S2).

Subsequently, as is described below with reference to FIG. 12, theconversion characteristic is calculated (step S3).

Furthermore, as is described below with reference to FIG. 13, the highfrequency component is separated into the invalid component caused bythe noise and the other valid component (step S4).

Then, as is described below with reference to FIG. 14, the gradationprocessing is performed on the low frequency component and the validcomponent in the high frequency component (step S5).

Next, on the basis of the low frequency component on which the gradationprocessing has been performed, the valid component in the high frequencycomponent on which the gradation processing has been performed, and theinvalid component in the high frequency component, the image signal onwhich the gradation conversion has been performed is synthesized (stepS6).

Subsequently, the signal processing such as a known compressionprocessing is performed (step S7).

Then, the image signal after the processing is output (step S8), and theprocessing is ended.

FIG. 12 is a flow chart showing the processing for the conversioncharacteristic calculation in the above-mentioned step S3.

When the processing is started, as illustrated in FIG. 7, the lowfrequency component is divided into regions of a predetermined size tobe sequentially extracted (step S10).

Next, the low frequency components are compared with the pre-setthreshold related to the dark part and the pre-set threshold related tothe light part respectively to extract the low frequency componentswhich are equal to or larger than the threshold of the dark part andalso equal to or smaller than the threshold of the light part as thecorrect exposure range (step S11).

Subsequently, by using the Laplacian filter with respect to the lowfrequency components in the correct exposure range, the knowncalculation for the edge intensity is performed (step S12).

Then, by selecting the pixels having the edge intensity equal to orlarger than the pre-set threshold, the histogram is created (step S13).

After that, by accumulating the histograms and further performing thenormalization, the gradation conversion curve is calculated (step S14).

The gradation conversion curve calculated in the above-mentioned manneris output (step S15).

Subsequently, it is determined whether or not the processing has beenperformed for all the regions (step S16). In a case where it isdetermined that the processing has not been completed, the flow isreturned to the above-mentioned step S10 to repeat the above-mentionedprocessing. On the other hand, in a case where it is determined that theprocessing has been completed, the flow is returned to the processingshown in FIG. 11.

FIG. 13 is a flow chart showing the processing for the high frequencyseparation in the above-mentioned step S4.

When the processing is started, first, the low frequency components aresequentially extracted for each pixel (step S20).

Next, from the read header information, the information such as thetemperature and the gain of the image pickup device is set. At thistime, if a necessary parameter does not exist for the headerinformation, a pre-set standard value is assigned to the relevantinformation (step S21).

Subsequently, the parameter related to the reference noise model is read(step S22).

Then, on the basis of the parameter of the reference noise model, thenoise amount related to the low frequency component is calculatedthrough the interpolation processing (step S23).

After that, as illustrated in FIG. 3B or 3C, the high frequencycomponent corresponding to the low frequency component and the highfrequency components located in the neighborhood of the high frequencycomponent are sequentially extracted (step S24).

Next, from the high frequency component corresponding to the lowfrequency component and the high frequency components located in theneighborhood of the high frequency component, the average value iscalculated (step S25).

Subsequently, on the basis of the average value and the noise amount,the upper limit and the lower limit are set as shown in NumericExpression 3 (step S26).

Then, in a case where the high frequency component is in the rangebetween the upper limit and the lower limit, it is determined that thehigh frequency component is the invalid component caused by the noise,and in a case where the high frequency component exceeds the upper limitor falls short of the lower limit, it is determined that the highfrequency component is the valid component (step S27).

Furthermore, the valid component and the invalid component are outputwhile being separated from each other (step S28).

Then, it is determined whether or not the processing for all the highfrequency components has been completed (step S29). In a case where itis determined that the processing has not been completed, the flow isreturned to the above-mentioned step S24 to repeat the above-mentionedprocessing.

On the other hand, in the step S29, in a case where it is determinedthat the processing for all the high frequency components has beencompleted, it is determined whether or not the processing for all thelow frequency components has been completed (step S30). In a case whereit is determined that the processing has not been completed, the flow isreturned to the above-mentioned step S20 to repeat the above-mentionedprocessing. On the other hand, in a case where it is determined that theprocessing has been completed, the flow is returned to the processingshown in FIG. 11.

FIG. 14 is a flow chart showing the processing for the gradationprocessing in the above-mentioned step S5.

When the processing is started, first, the low frequency components aresequentially extracted for each pixel (step S40).

Next, as illustrated in FIG. 8, the distances between the target pixelof the low frequency component and the centers of the four neighboringregions are calculated (step S41).

Subsequently, the gradation conversion curves in the four neighboringregions are read (step S42).

Furthermore, as shown in Numeric Expression 4, the gradation conversionequation with respect to the target pixel is set (step S43).

Then, as illustrated in FIG. 3B or 3C, the high frequency componentsregarded as the valid components corresponding to the low frequencycomponents are sequentially extracted (step S44).

After that, it is determined whether or not the high frequency componentregarded as the valid component exists (step S45).

At this time, in a case where it is determined that the high frequencycomponent regarded as the valid component exists, the gradationconversion equation shown in Numeric Expression 4 is applied to the highfrequency component regarded as the valid component to perform thegradation conversion (step S46).

When the processing in the step S46 is ended or in a case where it isdetermined that the high frequency component regarded as the validcomponent does not exist in the above-mentioned step S45, the gradationconversion equation shown in Numeric Expression 4 is applied to the lowfrequency component to perform the gradation conversion (step S47).

Then, the low frequency component on which the gradation processing hasbeen performed and the valid component in the high frequency componenton which the gradation processing has been performed are output (stepS48).

After that, it is determined whether or not the processing for all thelow frequency components has been completed (step S49). In a case whereit is determined that the processing has not been completed, the flow isreturned to the above-mentioned step S40 to repeat the above-mentionedprocessing. On the other hand, in a case where it is determined that theprocessing has been completed, the flow is returned to the processingshown in FIG. 11.

It should be noted that in the above, the configuration of using thewavelet transform for the frequency decomposition and the frequencysynthesis is adopted, but the configuration is not necessarily limitedto the above. For example, a configuration of using the known frequencydecomposition such as the Fourier transform, the discrete cosinetransform or the transform for the frequency synthesis can also beadopted.

Also, in the above, the number of times to perform the wavelet transformis set as two, but the configuration is not necessarily limited to theabove. For example, such a configuration can be adopted that byincreasing the number of times to perform the conversion, the separationof the invalid component caused by the noise and the other validcomponent is improved, or by decreasing the number of times to performthe conversion, the uniformity of the image is improved.

With the method of the space-invariant gradation processing using thesingle gradation conversion curve described above in the backgroundsection in the related art, in a non-standard situation such as abacklight, there is a problem that it is difficult to obtain anappropriate image signal.

Also, according to the technology disclosed in Japanese Patent No.3465226 described above in the background section, the gradationconversion curve is calculated for each image on the basis of thehistogram, but the increase in the noise components is not taken intoaccount. For this reason, for example, when a ratio of the dark part inthe image is large, the gradation conversion curve based on thehistogram provides a wide gradation to the dark part. However, in thiscase, the noise in the dark part prominently appears, and there is aproblem that an optimal gradation conversion processing is not performedin terms of image quality.

Furthermore, according to the technology disclosed in JapaneseUnexamined Patent Application Publication No. 8-56316 described above inthe background section, the contrast emphasis processing is performedonly on the low frequency component. Therefore, there is a problem thatthe sharpness is degraded in a region containing a large number of highfrequency components such as an edge region. Also, according to thetechnology disclosed in the publication, different processings areperformed on the low frequency component and other components.Therefore, there is a problem that the continuity and integrity for theimage as a whole may be lost.

Then, according to the technology disclosed in Japanese UnexaminedPatent Application Publication No. 2004-128985 described above in thebackground section, the noise reducing processing and other gradationprocessing are independent from each other. Therefore, there is aproblem that it is difficult to mutually utilize the processings in anoptimal manner.

In contrast with the above-mentioned background technology, according tothe first embodiment of the present invention, only the high frequencycomponent where the influence of the noise prominently visually appearsis separated into the invalid component and the valid component. Thegradation processing is performed on the valid component, and thegradation processing is not performed on the invalid component, and anincrease in noise accompanying with the gradation processing issuppressed. Thus, it is possible to generate the high quality imagesignal.

Also, as the low frequency component is excluded from the target of theprocessing after being separated into the valid component and theinvalid component, a possibility of generating an adverse effectaccompanying with the processing is decreased, and it is possible toimprove the stability.

Furthermore, as the image signal is synthesized with the invalidcomponent, it is possible to obtain the image signal with little senseof visual discomfort, and the stability and reliability of theprocessing can be improved.

Also, the wavelet transform is excellent at the separation of thefrequency, and it is therefore possible to perform the high accuracyprocessing.

As the gradation conversion curve is adaptively and also independentlycalculated for each region from the low frequency component of the imagesignal, it is possible to perform the gradation conversion at the highaccuracy on various image signals.

Also, as the gradation conversion curve is calculated on the basis ofthe low frequency component, it is possible to calculate the appropriategradation conversion curve with little influence from the noise.

As the gradation conversion with the identical conversion characteristicis performed on the low frequency component and the valid component inthe high frequency component located at the same position, it ispossible to obtain the image signal providing the sense of integritywith little sense of visual discomfort.

Also, as the gradation conversion curves independently obtained for eachregion are synthesized to set the gradation conversion equation used forthe gradation conversion of the target pixel, the discontinuity betweenthe regions is not generated, and it is possible to obtain the highquality image signals.

Then, in a case where the valid component in the high frequencycomponent does not exist, the unnecessary gradation conversion iscancelled, and it is thus possible to improve the processing speed.

Second Embodiment

FIGS. 15 to 25 illustrate a second embodiment of the present invention,and FIG. 15 is a block diagram of a configuration of an image processingsystem.

According to the second embodiment, the same part as that of the firstembodiment described above is allocated with the same name and referencenumeral to appropriately omit a description thereof, and only adifferent part will be mainly described.

The image processing system according to the present embodiment has aconfiguration in which with respect to the above-mentioned imageprocessing system illustrated in FIG. 1 according to the firstembodiment, a pre-white balance unit 801, a Y/C separation unit 802constituting Y/C separation means, a buffer 803, and a Y/C synthesisunit 809 constituting Y/C synthesis means are added, and the CCD 102,the frequency decomposition unit 109, the conversion characteristiccalculation unit 111, the high frequency separation unit 112, thegradation processing unit 113, and the frequency synthesis unit 115 arereplaced by a color CCD 800, a frequency decomposition unit 804constituting separation means and frequency decomposition means, aconversion characteristic calculation unit 805 constituting conversionmeans and conversion characteristic calculation means, a high frequencyseparation unit 806 constituting separation means and high frequencyseparation means, a gradation processing unit 807 constitutingconversion means and gradation processing means, and a frequencysynthesis unit 808 constituting synthesis means and frequency synthesismeans. Other basic configuration is similar to that of theabove-mentioned first embodiment. Therefore, the same components areallocated with the same names and reference numerals to appropriatelyomit the description thereof, and only a different part will be mainlydescribed.

The color image signal captured via the lens system 100, the aperture101, and the color CCD 800 is transferred to the amplification unit 103.

The buffer 105 is connected to the exposure control unit 106, the focuscontrol unit 107, the pre-white balance unit 801, and the Y/C separationunit 802.

The pre-white balance unit 801 is connected to the amplification unit103.

The Y/C separation unit 802 is connected to the buffer 803, and thebuffer 803 is connected to the frequency decomposition unit 804, theconversion characteristic calculation unit 805, and the Y/C synthesisunit 809.

The frequency decomposition unit 804 is connected to the buffer 110. Thebuffer 110 is connected to the conversion characteristic calculationunit 805, the high frequency separation unit 806, and the gradationprocessing unit 807. The conversion characteristic calculation unit 805is connected to the gradation processing unit 807. The high frequencyseparation unit 806 is connected to the buffer 114 and the gradationprocessing unit 807. The gradation processing unit 807 is connected tothe buffer 114.

The buffer 114 is connected via the frequency synthesis unit 808 and theY/C synthesis unit 809 to the signal processing unit 116.

The control unit 118 is also bi-directionally connected to the pre-whitebalance unit 801, the Y/C separation unit 802, the frequencydecomposition unit 804, the conversion characteristic calculation unit805, the high frequency separation unit 806, the gradation processingunit 807, the frequency synthesis unit 808, and the Y/C synthesis unit809 to control these units.

Also, the temperature sensor 120 according to the present embodiment isarranged in a neighborhood of the color CCD 800, and the signal from thetemperature sensor 120 is also connected to the control unit 118.

Next, the action of the image processing system illustrated in FIG. 15is basically similar to that of the first embodiment, and therefore onlya different part will be mainly described along the flow of the imagesignal.

When the user performs a half press of the shutter button which iscomposed of a two-stage switch of the external I/F unit 119, the imageprocessing system functions as the pre-image pickup device.

After that, the color image signal captured via the lens system 100, theaperture 101, and the color CCD 800 is transferred via the amplificationunit 103 and the A/D conversion unit 104 to the buffer 105. It should benoted that according to the present embodiment, as the color CCD 800, asingle CCD in which a Bayer-type primary color filter is arranged on afront face is supposed.

Herein, FIG. 16 is a diagram illustrating a configuration of theBayer-type primary color filter.

The Bayer-type primary color filter has a such configuration that thatthe basic unit is 2×2 pixels, one each of a red (R) filter and a blue(B) filter are arranged at pixel positions at opposite corners in thebasis unit, and green (G) filters are arranged at pixel positions atremaining opposite corners.

Subsequently, the color image signal in the buffer 105 is transferred tothe pre-white balance unit 801. The pre-white balance unit 801multiplies signals at a predetermined level for each color signal (inother words, cumulatively adds) to calculate a simplified white balancecoefficient. The pre-white balance unit 801 transfers the calculatedcoefficient to the amplification unit 103 and multiplies different gainsfor each color signal to perform the white balance.

In this way, when the focus adjustment, the exposure adjustment, thesimplified white balance adjustment, and the like are performed, theuser performs the full press of the shutter button composed of thetwo-stage switch of the external I/F unit 119. After that, the digitalcamera functions as the real shooting device.

After that, similarly to the pre shooting, the color image signal istransferred to the buffer 105. The white balance coefficient calculatedby the pre-white balance unit 801 at this time is transferred to thecontrol unit 118.

The color image signal in the buffer 105 obtained through the realshooting operation is transferred to the Y/C separation unit 802.

On the basis of the control of the control unit 118, through a knowninterpolation processing, the Y/C separation unit 802 generates thethree color image signals composed of R, G, and B, and further separatesthe R, G, and B signals into a luminance signal Y and color differencesignals Cb and Cr as shown in Numeric Expression 5 below.Y=0.29900R+0.58700G+0.11400BCb=−0.16874R−0.33126G+0.50000BCr=0.50000R−0.41869G−0.08131B  [Expression 5]

The luminance signal and the color difference signals separated by theY/C separation unit 802 are transferred to the buffer 803.

On the basis of the control of the control unit 118, the frequencydecomposition unit 804 performs the frequency decomposition on theluminance signal in the buffer 105, and the high frequency component andthe low frequency component are obtained. Then, the frequencydecomposition unit 804 sequentially transfers the high frequencycomponent and the low frequency component thus obtained to the buffer110.

The conversion characteristic calculation unit 805 reads the lowfrequency component from the buffer 110 from on the basis of the controlof the control unit 118, and the color difference signals from thebuffer 803, respectively, to calculate the gradation characteristic usedfor the gradation conversion processing. It should be noted thataccording to the present embodiment, as the gradation conversionprocessing, the space-invariant processing using the single gradationconversion curve is supposed with respect to the image signal. Then, theconversion characteristic calculation unit 805 transfers the calculatedgradation characteristics to the gradation processing unit 807.

The high frequency separation unit 806 reads the high frequencycomponent from the buffer 110 and the high frequency component isseparated into the invalid component caused by the noise and the othervalid component on the basis of the control of the control unit 118.Then, the high frequency separation unit 806 transfers the thusseparated valid component to the gradation processing unit 807 and theabove-mentioned invalid component to the buffer 114, respectively.

The gradation processing unit 807 reads the low frequency component fromthe buffer 110, the valid components in the high frequency componentfrom the high frequency separation unit 806, and the gradationcharacteristic from the conversion characteristic calculation unit 805,respectively, on the basis of the control of the control unit 118. Then,on the basis of the above-mentioned gradation characteristic, thegradation processing unit 807 performs the gradation processing on thelow frequency component and the valid component in the high frequencycomponent. The gradation processing unit 807 transfers the low frequencycomponent on which the gradation processing has been performed and thevalid component in the high frequency component on which the gradationprocessing has been performed to the buffer 114.

The frequency synthesis unit 808 reads the low frequency component onwhich the gradation processing has been performed, the valid componentin the high frequency component on which the gradation processing hasbeen performed, and the invalid component in the high frequencycomponent from the buffer 114 and performs an additional processing onthe basis of these components to synthesize the luminance signals onwhich the gradation conversion has been performed with each other on thebasis of the control of the control unit 118. Then, the frequencysynthesis unit 808 transfers the synthesized luminance signal to the Y/Csynthesis unit 809.

The Y/C synthesis unit 809 reads the luminance signal Y′ on which thegradation conversion has been performed from the frequency synthesisunit 808 and the color difference signals Cb and Cr from the buffer 803,respectively, to synthesize color image signals R′, G′, and B′ on whichthe gradation conversion has been performed as shown in NumericExpression 6 below on the basis of the control of the control unit 118.R′=Y′+1.40200CrG′=Y′−0.34414Cb−0.71414CrB′=Y′+1.77200Cb  [Expression 6]

The Y/C synthesis unit 809 transfers the synthesized color image signalsR′, G′, and B′ to the signal processing unit 116.

The signal processing unit 116 performs a known compression processingor the like on the image signal from the Y/C synthesis unit 809 andtransfers the signal after the processing to the output unit 117 on thebasis of the control of the control unit 118.

The output unit 117 records and saves the image signal output from thesignal processing unit 116 in the recording medium such as a memorycard.

Next, FIG. 18 is a block diagram of a configuration example of thefrequency decomposition unit 804.

The frequency decomposition unit 804 is configured by including a signalextraction unit 900, a low-pass filter unit 901, a low frequency buffer902, and a difference filter unit 903.

The buffer 803 is connected to the signal extraction unit 900. Thesignal extraction unit 900 is connected to the low-pass filter unit 901and the difference filter unit 903. The low-pass filter unit 901 isconnected to the low frequency buffer 902. The low frequency buffer 902is connected to the difference filter unit 903. The difference filterunit 903 is connected to the buffer 110.

The control unit 118 is bi-directionally connected to the signalextraction unit 900, the low-pass filter unit 901, and the differencefilter unit 903 to control these units.

Subsequently, a description will be given of the action of the frequencydecomposition unit 804.

The signal extraction unit 900 reads the luminance signals from thebuffer 803 on the basis of the control of the control unit 118 totransfer the luminance signals to the low-pass filter unit 901 and thedifference filter unit 903.

The low-pass filter unit 901 performs a known low-pass filter processingon the luminance signals from the signal extraction unit 900 tocalculate the low frequency components of the luminance signals on thebasis of the control of the control unit 118. It should be noted thataccording to the present embodiment, as the low-pass filter used by thelow-pass filter unit 901, for example, an average value filter having apixel size of 7×7. The low-pass filter unit 901 transfers the calculatedlow frequency components to the low frequency buffer 902.

The difference filter unit 903 reads the luminance signals from thesignal extraction unit 900 and the low frequency components of theluminance signals from the low frequency buffer 902, respectively, andtakes a difference thereof to calculate the high frequency components ofthe luminance signals. The difference filter unit 903 transfers thecalculated high frequency components and the read low frequencycomponents to the buffer 110.

Next, FIG. 19 is a block diagram of a configuration example of theconversion characteristic calculation unit 805.

The conversion characteristic calculation unit 805 has such aconfiguration that with respect to the conversion characteristiccalculation unit 111 shown in FIG. 4 of the above-mentioned firstembodiment, a hue calculation unit 1000 constituting region-of-interestsetting means, a person determination unit 1001 constitutingregion-of-interest setting means, a weighting factor setting unit 1002constituting weighting factor setting means, and a histogram correctionunit 1003 constituting histogram correction means are added, and thedivision unit 300 and the buffer 301 are omitted. Other basicconfiguration is similar to that of the conversion characteristiccalculation unit 111 shown in FIG. 4. Therefore, the same components areallocated with the same names and reference numerals to appropriatelyomit the description thereof, and only a different part will be mainlydescribed.

The buffer 803 and the buffer 110 are connected to the correct rangeextraction unit 302. The correct range extraction unit 302 is connectedto the edge calculation unit 303 and the hue calculation unit 1000.

The hue calculation unit 1000 is connected via the person determinationunit 1001 and the weighting factor setting unit 1002 to the histogramcorrection unit 1003.

The histogram creation unit 304 is connected to the histogram correctionunit 1003.

The histogram correction unit 1003 is connected via the gradationconversion curve calculation unit 305 and the buffer 306 to thegradation processing unit 807.

The control unit 118 is also bi-directionally connected to the huecalculation unit 1000, the person determination unit 1001, the weightingfactor setting unit 1002, and the histogram correction unit 1003 tocontrol these units.

Subsequently, a description will be given of the action of theconversion characteristic calculation unit 805.

The correct range extraction unit 302 reads the luminance signals fromthe buffer 110 which are compared with the pre-set threshold related tothe dark part (by way of an example, in the case of 12-bit gradation,for example, 128) and the pre-set threshold related to the light part(in the case of the 12-bit gradation, for example, 3968) respectively,and transfers the luminance signals which are equal to or larger thanthe threshold of the dark part and also equal to or smaller than thethreshold of the light part as the correct exposure range to the edgecalculation unit 303 on the basis of the control of the control unit118.

Also, the correct range extraction unit 302 reads the color differencesignals Cb and Cr at coordinates corresponding to the luminance signalsin the correct exposure range from the buffer 803 to be transferred tothe hue calculation unit 1000.

The edge calculation unit 303 and the histogram creation unit 304 createthe histogram of edge regions from the luminance signals similarly tothe above-mentioned first embodiment, and transfer the created histogramto the histogram correction unit 1003.

The hue calculation unit 1000 reads the color difference signals Cb andCr from the correct range extraction unit 302 which are compared withthe pre-set threshold to extract a skin color region, and transfers theresult to the person determination unit 1001 on the basis of the controlof the control unit 118.

The person determination unit 1001 uses the information related to theskin color region from the hue calculation unit 1000 and the edge amountfrom the edge calculation unit 303 to extract a region determined as ahuman face, and transfers the result to the weighting factor settingunit 1002 on the basis of the control of the control unit 118.

On the basis of the control of the control unit 118, the weightingfactor setting unit 1002 calculates luminance information in the regiondetermined as the human face which is multiplied by a predeterminedcoefficient, thereby weighting factors for the corrections at therespective luminance levels are calculated. It should be noted that theweighting factors at the luminance levels which do not exist in theregion determined as the human face are 0. The weighting factor settingunit 1002 transfers the calculated weighting factors to the histogramcorrection unit 1003.

The histogram correction unit 1003 reads the histogram from thehistogram creation unit 304 and also reads the weighting factors fromthe weighting factor setting unit 1002 on the basis of the control ofthe control unit 118. Then, the histogram correction unit 1003 adds theweighting factors to the respective luminance levels of the histogram toperform the correction. The corrected histogram is transferred to thegradation conversion curve calculation unit 305, and similarly to theabove-mentioned first embodiment, the gradation conversion curve iscalculated.

The calculated gradation conversion curve is transferred to the buffer306, and when necessary, transferred to the gradation processing unit807. It should be noted that according to the present embodiment, thespace-invariant processing is supposed, and the calculated gradationconversion curve is of one type.

Next, FIG. 20 is a block diagram of a configuration example of the highfrequency separation unit 806.

The high frequency separation unit 806 has such a configuration thatwith respect to the high frequency separation unit 112 shown in FIG. 5of the above-mentioned first embodiment, a noise LUT 1100 constitutingnoise estimation means and table conversion means are added, and theparameter ROM 403, the parameter selection unit 404, and theinterpolation unit 405 are omitted. Other basic configuration is similarto that of the high frequency separation unit 112 shown in FIG. 5.Therefore, the same components are allocated with the same names andreference numerals to appropriately omit the description thereof, andonly a different part will be mainly described.

The low frequency component extraction unit 400, the gain calculationunit 401, and the standard value assigning unit 402 are connected to thenoise LUT 1100. The noise LUT 1100 is connected to the upper limit andlower limit setting unit 408.

The determination unit 409 is connected to the gradation processing unit807 and the buffer 114.

The control unit 118 is also bi-directionally connected to the noise LUT1100 to control the table.

Subsequently, a description will be given of the action of the highfrequency separation unit 806.

The gain calculation unit 401 calculates the gain information in theamplification unit 103 which is transferred to the noise LUT 1100 on thebasis of the ISO sensitivity, the information related to the exposureconditions, and the white balance coefficient sent from the control unit118.

Also, the control unit 118 obtains temperature information of the colorCCD 800 from the temperature sensor 120 and transfers the thus obtainedtemperature information to the noise LUT 1100.

On the basis of the control of the control unit 118, in a case where atleast one of the above-mentioned gain information and the temperatureinformation cannot be obtained, the standard value assigning unit 402transfers a standard value of the information that cannot be obtained tothe noise LUT 1100.

The noise LUT 1100 is a look up table where a relation among the signalvalue level of the image signal, the gain of the image signal, and theoperation temperature of the image pickup device, and the noise amountis recorded. The look up table is designed, for example, by using thetechnology disclosed in Japanese Unexamined Patent ApplicationPublication No. 2004-128985 described above. The noise LUT 1100 outputsthe noise amount on the basis of the pixel value of the target pixelfrom the low frequency component extraction unit 400, the gaininformation from the gain calculation unit 401 or the standard valueassigning unit 402, and the temperature information from the controlunit 118 or the standard value assigning unit 402. The output noiseamount is transferred to the upper limit and lower limit setting unit408.

The high frequency component extraction unit 406 extracts the highfrequency component corresponding to the low frequency componentextracted by the low frequency component extraction unit 400 and thehigh frequency components located in the neighborhood of the highfrequency component on the basis of the control of the control unit 118.

It should be noted that according to the present embodiment, asdescribed above, the frequency decomposition unit 804 uses the low-passfilter and the difference filter to extract the low frequency componentand the high frequency component. Therefore, the pixel configurations ofthe low frequency component and the high frequency component are of thesame size and, and the high frequency component corresponding to the lowfrequency component is one pixel.

The action of the high frequency separation unit 806 thereafter issimilar to that of the high frequency separation unit 112 of theabove-mentioned first embodiment. The high frequency component isseparated into the valid component and the invalid component. The validcomponent is transferred to the gradation processing unit 807, and theinvalid component is transferred to the buffer 114, respectively.

Next, FIG. 21 is a block diagram of a configuration example of thegradation processing unit 807.

The gradation processing unit 807 has such a configuration that withrespect to the gradation processing unit 113 shown in FIG. 6 of theabove-mentioned first embodiment, the distance calculation unit 501, thegradation conversion equation setting unit 502, and the buffer 503 aredeleted. Other basic configuration is similar to that of the gradationprocessing unit 113 shown in FIG. 6. Therefore, the same components areallocated with the same names and reference numerals to appropriatelyomit the description thereof, and only a different part will be mainlydescribed.

The conversion characteristic calculation unit 805 is connected to thegradation conversion unit 505.

The buffer 110 is connected via the low frequency component extractionunit 500 to the gradation conversion unit 505. The high frequencyseparation unit 806 is connected via the high frequency componentextraction unit 504 to the gradation conversion unit 505.

The control unit 118 is bi-directionally connected to the low frequencycomponent extraction unit 500, the high frequency component extractionunit 504, and the gradation conversion unit 505 to control these units.

Subsequently, a description will be given of the action of the gradationprocessing unit 807.

The low frequency component extraction unit 500 sequentially extractsthe low frequency components from the buffer 110 for each pixel on thebasis of the control of the control unit 118. The low frequencycomponent extraction unit 500 transfers the extracted low frequencycomponents to the gradation conversion unit 505.

The high frequency component extraction unit 504 extracts the highfrequency components corresponding to the low frequency componentsextracted by the low frequency component extraction unit 500 from thehigh frequency separation unit 806 on the basis of the control of thecontrol unit 118. According to the present embodiment, as describedabove, the pixel configurations of the low frequency component and thehigh frequency component are of the same size, and the high frequencycomponent corresponding to the low frequency component is one pixel. Itshould be noted that in a case where it is determined that the highfrequency component corresponding to the low frequency component is theinvalid component and the extracted high frequency component does notexist, the high frequency component extraction unit 504 transfers theerror information to the control unit 118.

The gradation conversion unit 505 reads the low frequency componentsfrom the low frequency component extraction unit 500 on the basis of thecontrol of the control unit 118 and reads the gradation conversion curvefrom the conversion characteristic calculation unit 805 to perform thegradation conversion on the low frequency components. The gradationconversion unit 505 transfers the low frequency component after thegradation conversion to the buffer 114.

After that, the gradation conversion unit 505 reads the high frequencycomponent of the valid component corresponding to the low frequencycomponent from the high frequency component extraction unit 504 toperform the gradation conversion. Then, the gradation conversion unit505 transfers the high frequency component after the gradationconversion to the buffer 114. It should be noted that in a case wherethe high frequency component corresponding to the low frequencycomponent does not exist, the gradation conversion unit 505 cancel thegradation conversion on the high frequency component on the basis of thecontrol of the control unit 118.

It should be noted that according to the present embodiment too,similarly to the above-mentioned first embodiment, the image processingsystem in which the image pickup unit is separately provided may beused.

Also, in the above, it is supposed to perform the processing by way ofthe hardware, but the configuration is not necessarily limited to theabove. For example, the color image signal from the color CCD 800 isrecorded on the recording medium such as a memory card as raw data whilebeing unprocessed, and the associated information such as image pickupconditions (for example, the temperature of the image pickup device, theexposure conditions, and the like, for each shooting operation from thecontrol unit 118) is recorded in the recording medium as the headerinformation. Then, the processing can be performed as the computer isallowed to execute the image processing program which is separatesoftware to instruct the computer to read the information of therecording medium. It should be noted that the transmission of variouspieces of information from the image pickup unit to the computer is notnecessarily performed via the recording medium and may be performed viaa communication line or the like.

FIG. 22 is a flow chart showing a main routine of an image processingprogram.

It should be noted that in FIG. 22, processing steps basicallysubstantially identified with the processing shown in FIG. 11 of theabove-mentioned first embodiment are allocated with the same stepnumbers.

When the processing is started, first, the color image signal is read,and also the header information such as the temperature and the exposureconditions of the image pickup device is read (step S1).

Next, as shown in Numeric Expression 5, the luminance signals and thecolor difference signals are calculated (step S50).

Subsequently, by using the low-pass filter and the difference filter,the frequency decomposition on the luminance signals is performed, andthe high frequency component and the low frequency component areobtained (step S2).

Furthermore, as is described below with reference to FIG. 23, theconversion characteristic is calculated (step S51).

Then, as is described below with reference to FIG. 24, the highfrequency component is separated into the invalid component caused bythe noise and the other valid component (step S52).

Next, as is described below with reference to FIG. 25, the gradationprocessing is performed on the low frequency component and the validcomponent in the high frequency component (step S53).

Subsequently, on the basis of the low frequency component on which thegradation processing has been performed, the valid component in the highfrequency component on which the gradation processing has beenperformed, and the invalid component in the high frequency component,the luminance signals on which the gradation conversion has beenperformed are synthesized one another (step S6).

Then, as shown in Numeric Expression 6, the luminance signals and thecolor difference signals are synthesized to obtain the color imagesignal on which the gradation conversion has been performed (step S54).

Furthermore, the signal processing such as a known compressionprocessing is performed (step S7).

After that, the color image signal after the processing is output (stepS8), and the processing is ended.

FIG. 23 is a flow chart showing the processing for the conversioncharacteristic calculation in the above-mentioned step S51.

It should be noted that in FIG. 23, processing steps basicallysubstantially identified with the processing shown in FIG. 12 of theabove-mentioned first embodiment are allocated with the same stepnumbers.

When the processing is started, the luminance signals are compared withthe pre-set threshold related to the dark part and the pre-set thresholdrelated to the light part to extract the luminance signals which areequal to or larger than the threshold of the dark part and also equal toor smaller than the threshold of the light part as the correct exposurerange (step S11).

Subsequently, the known calculation for the edge intensity is performedon the luminance signals in the correct exposure range by using theLaplacian filter or the like (step S12).

Then, by selecting the pixels having the edge intensity equal to orlarger than the pre-set threshold, the histogram is created (step S13).

After that, by comparing the color difference signal with the pre-setthreshold, a particular hue region, for example, a skin color region isextracted (step S60).

Furthermore, on the basis of the skin color region and the informationon the edge intensity, the region determined as the human face isextracted and set as a region-of-interest (step S61).

Next, the luminance information in the region-of-interest is calculatedand multiplied by a pre-set coefficient to calculate the weightingfactors for the correction related to the respective luminance levels(step S62).

Subsequently, the weighting factors are added to the respectiveluminance levels of the histogram to perform the correction on thehistogram (step S63).

After that, by accumulating the histograms and further performing thenormalization, the gradation conversion curve is calculated (step S14).

The gradation conversion curve calculated in the above-mentioned manneris output (step S15), and the flow is returned from the processing tothe processing shown in FIG. 22.

FIG. 24 is a flow chart showing the processing for the high frequencyseparation.

It should be noted that in FIG. 24, processing steps basicallysubstantially identified with the processing shown in FIG. 13 of theabove-mentioned first embodiment are allocated with the same stepnumbers.

When the processing is started, first, the low frequency components aresequentially extracted for each pixel (step S20).

Next, from the read header information, the information such as thetemperature and the gain of the image pickup device is set. At thistime, if a necessary parameter does not exist for the headerinformation, a pre-set standard value is assigned to the relevantinformation (step S21).

Subsequently, the table related to the noise amount where a relationamong the signal value level of the image signal, the gain of the imagesignal, the operation temperature of the image pickup device, and thenoise amount is recorded is read (step S70).

Then, on the basis of the table related to the noise amount, the noiseamount is calculated (step S71).

After that, the high frequency component corresponding to the lowfrequency component and the high frequency components located in theneighborhood of the high frequency component are extracted (step S24).

Furthermore, from the high frequency component corresponding to the lowfrequency component and the high frequency components located in theneighborhood of the high frequency component, the average value iscalculated (step S25).

Next, on the basis of the average value and the noise amount, the upperlimit and the lower limit are set as shown in Numeric Expression 3 (stepS26).

Subsequently, in a case where the high frequency component is in therange between the upper limit and the lower limit, it is determined thatthe high frequency component is the invalid component caused by thenoise, and in a case where the high frequency component exceeds theupper limit or falls short of the lower limit, it is determined that thehigh frequency component is the valid component (step S27).

Then, the valid component and the invalid component are output whilebeing separated from each other (step S28).

Furthermore, it is determined whether or not the processing for all thelow frequency components has been completed (step S30). In a case whereit is determined that the processing has not been completed, the flow isreturned to the above-mentioned step S20 to repeat the above-mentionedprocessing. On the other hand, in a case where it is determined that theprocessing has been completed, the flow is returned to the processingshown in FIG. 22.

FIG. 25 is a flow chart showing the gradation processing in theabove-mentioned step S53.

It should be noted that in FIG. 25, processing steps basicallysubstantially identified with the processing shown in FIG. 14 of theabove-mentioned first embodiment are allocated with the same stepnumbers.

When the processing is started, first, the low frequency components aresequentially extracted for each pixel (step S40).

Next, the gradation conversion curve is read (step S42).

Subsequently, the high frequency component regarded as the validcomponent corresponding to the low frequency component is extracted(step S44).

Then, it is determined whether or not the high frequency componentregarded as the valid component exists (step S45).

At this time, in a case where it is determined that the high frequencycomponent regarded as the valid component exists, the gradationconversion is performed on the high frequency component regarded as thevalid component (step S46).

When the processing in the step S46 is ended or in a case where it isdetermined that the high frequency component regarded as the validcomponent does not exist in the above-mentioned step S45, the gradationconversion is performed on the low frequency components (step S47).

Next, the low frequency component on which the gradation processing hasbeen performed and the valid component in the high frequency componenton which the gradation processing has been performed are output (stepS48).

After that, it is determined whether or not the processing for all thelow frequency components has been completed (step S49). In a case whereit is determined that the processing has not been completed, the flow isreturned to the above-mentioned step S40 to repeat the above-mentionedprocessing. On the other hand, in a case where it is determined that theprocessing has been completed, the flow is returned to the processingshown in FIG. 22.

It should be noted that in the above description, the configuration ofusing the low-pass filter and the difference filter for the frequencydecomposition and the frequency synthesis is adopted, but theconfiguration is not necessarily limited to the above. For example, aconfiguration of using a Gaussian filter and the Laplacian filter forthe frequency decomposition and the frequency synthesis may also beadopted. In this case, although the operation amount is increased, anadvantage is provided that the performance of the frequencydecomposition is better. Then, in a case where the Gaussian filter andthe Laplacian filter are used, similarly to the above-mentioned firstembodiment, a configuration of performing the frequency decompositionand the frequency synthesis at multi stages can be adopted.

Also, in the above description, for the color image pickup device, theconfiguration of using the Bayer-type primary color filter is adopted,but the configuration is not necessarily limited to the above. Forexample, the single image pickup device using a color-differenceline-sequential type complementary color filter shown in FIG. 17 or thetwo or three image pickup device may also be applied.

Herein, FIG. 17 is a diagram illustrating the configuration of thecolor-difference line-sequential type complementary color filter.

The color-difference line-sequential type complementary color filter hasa basic unit of 2×2 pixels. Cyan (Cy) and yellow (Ye) are arranged onthe same line of the 2×2 pixels, and magenta (Mg) and green (G) arearranged on the other line of the 2×2 pixels. It should be noted thatsuch a configuration is adopted that the positions of magenta (Mg) andgreen (G) are inverted for each line.

According to the second embodiment described above, only the highfrequency component where the influence of the noise prominentlyvisually appears with respect to the color signal is separated into theinvalid component and the valid component. The gradation processing isperformed on the valid component, and the gradation processing is notperformed on the invalid component, and an increase in noiseaccompanying with the gradation processing is suppressed. Thus, it ispossible to generate the high quality color image signal.

Also, as the low frequency component is excluded from the target of theprocessing after being separated into the valid component and theinvalid component, the possibility of generating the adverse effectaccompanying with the processing is decreased, and it is possible toimprove the stability.

Furthermore, as the image signal is synthesized with the invalidcomponent, it is possible to obtain the color image signal with littlesense of visual discomfort, and the stability and reliability of theprocessing can be improved.

Then, as the low-pass filter and the difference filter has a simplefilter configuration, the image processing system in which theprocessing can be performed at a high speed can be configured at a lowcost.

In addition, as the gradation conversion curve is obtained adaptivelyfrom the low frequency components of the luminance signals, it ispossible to perform the high accuracy gradation conversion on varioustypes of the color image signals.

Also, as the gradation conversion curve is calculated on the basis ofthe low frequency component, it is possible to calculate the appropriategradation conversion curve with little influence from the noise.

Furthermore, as the gradation processing can be performed whileweighting the region-of-interest such as a human being, it is possibleto obtain the high quality image signals which are subjectivelypreferable.

Then, as the gradation conversion with the identical conversioncharacteristic is performed on the low frequency component and the validcomponent in the high frequency component located at the same position,it is possible to obtain the image signal providing the sense ofintegrity with little sense of visual discomfort.

In addition, in a case where the valid component in the high frequencycomponent does not exist, the unnecessary gradation conversion iscancelled, and it is thus possible to improve the processing speed.

Third Embodiment

FIGS. 26 to 30 illustrate a third embodiment of the present invention,and FIG. 26 is a block diagram of a configuration of an image processingsystem.

According to the third embodiment, a part similar to that of theabove-mentioned first and second embodiments is allocated with the samereference numerals to appropriately omit the description thereof, andonly a different part will be mainly described.

The image processing system according to the present embodiment has sucha configuration that with respect to the above-mentioned imageprocessing system illustrated in FIG. 1 according to the firstembodiment, an edge emphasis unit 1202 constituting edge emphasis meansis added, and the frequency decomposition unit 109, the high frequencyseparation unit 112, and the frequency synthesis unit 115 arerespectively replaced by a frequency decomposition unit 1200constituting separation means and frequency decomposition means, highfrequency separation unit 1201 constituting separation means and highfrequency separation means, and a frequency synthesis unit 1203constituting synthesis means and frequency synthesis means. Other basicconfiguration is similar to that of the above-mentioned firstembodiment. Therefore, the same components are allocated with the samenames and reference numerals to appropriately omit the descriptionthereof, and only a different part will be mainly described.

The buffer 105 is connected to the exposure control unit 106, the focuscontrol unit 107, the conversion characteristic calculation unit 111,and the frequency decomposition unit 1200.

The frequency decomposition unit 1200 is connected to the buffer 110.The buffer 110 is connected to the conversion characteristic calculationunit 111, the high frequency separation unit 1201, and the gradationprocessing unit 113.

The high frequency separation unit 1201 is connected to the edgeemphasis unit 1202 and the buffer 114. The edge emphasis unit 1202 isconnected to the gradation processing unit 113.

The buffer 114 is connected via the frequency synthesis unit 1203 to thesignal processing unit 116.

The control unit 118 is also bi-directionally connected to the frequencydecomposition unit 1200, the high frequency separation unit 1201, theedge emphasis unit 1202, and the frequency synthesis unit 1203 tocontrol these units.

Next, the action of the image processing system illustrated in FIG. 26is basically similar to that of the first embodiment, and therefore onlya different part will be mainly described along the flow of the imagesignal.

The image signal in the buffer 105 is transferred to the frequencydecomposition unit 1200.

The frequency decomposition unit 1200 performs a predetermined frequencydecomposition on the transferred image signal to obtain a high frequencycomponent and a low frequency component on the basis of the control ofthe control unit 118. Then, the frequency decomposition unit 1200sequentially transfers the thus obtained high frequency component andthe low frequency components to the buffer 110. It should be noted thataccording to the present embodiment, for the frequency decomposition,for example, it is supposed to use a known discrete cosine transform ofa 64×64 pixel unit.

FIGS. 27A and 27B are explanatory diagrams for describing the discretecosine transform; FIG. 27A illustrates the image signal in the realspace and FIG. 27B illustrates the signal in the frequency space afterthe discrete cosine transform, respectively.

In the frequency space of FIG. 27B, the upper left is set as the origin,that is, as the zero-th order component, and the high frequencycomponents at the first-order or above are arranged on a concentriccircle while using the zero-th order component as the origin.

The conversion characteristic calculation unit 111 reads the imagesignal from the buffer 105 for each 64×64 pixel unit used in thefrequency decomposition unit 1200 on the basis of the control of thecontrol unit 118. After that, the conversion characteristic calculationunit 111 calculates the gradation characteristic used for the gradationconversion processing similarly to the above-mentioned first embodiment.That is, according to the present embodiment, for the gradationconversion processing, it is supposed to employ the space-variantprocessing using a plurality of gradation characteristics different foreach region at the 64×64 pixel unit. Then, the conversion characteristiccalculation unit 111 transfers the calculated the gradationcharacteristic to the gradation processing unit 113.

The high frequency separation unit 1201 reads the high frequencycomponents from the buffer 110 and performs the noise reducingprocessing on the high frequency components on the basis of the controlof the control unit 118. After that, the high frequency component isseparated into the invalid component caused by the noise and the othervalid component. Then, the high frequency separation unit 1201 transfersthe thus separated valid components to the edge emphasis unit 1202 andthe above-mentioned invalid components to the buffer 114, respectively.

The edge emphasis unit 1202 multiplies the valid component transferredby the high frequency separation unit 1201 by a pre-set coefficient toperform the edge emphasis processing, and transfers the processingresult to the gradation processing unit 113.

The gradation processing unit 113 reads the low frequency componentsfrom the buffer 110, the valid components in the high frequencycomponents from the edge emphasis unit 1202, and the gradationcharacteristic from the conversion characteristic calculation unit 111,respectively, on the basis of the control of the control unit 118. Then,on the basis of the above-mentioned gradation characteristic, thegradation processing unit 113 performs the gradation processing on thelow frequency component and the valid components in the high frequencycomponents. The gradation processing unit 113 transfers the lowfrequency component on which the gradation processing has been performedand the valid components in the high frequency components to the buffer114.

The frequency synthesis unit 1203 reads the low frequency component onwhich the gradation processing has been performed, the valid componentin the high frequency component on which the gradation processing hasbeen performed, and the invalid component in the high frequencycomponent from the buffer 114 on the basis of the control of the controlunit 118, and synthesizes the image signal on which the gradationprocessing has been performed on the basis of these components. Itshould be noted that according to the present embodiment, for thefrequency synthesis, it is supposed to use a known inverse DCT (DiscreteCosine Transform). Then, the frequency synthesis unit 1203 transfers thesynthesized image signal to the signal processing unit 116.

The signal processing unit 116 performs a known compression processingor the like on the image signal from the frequency synthesis unit 1203and transfers the signal after the processing to the output unit 117 onthe basis of the control of the control unit 118.

The output unit 117 records and saves the image signal output from thesignal processing unit 116 in the recording medium such as a memorycard.

Next, FIG. 28 is a block diagram of a configuration example of the highfrequency separation unit 1201.

The high frequency separation unit 1201 has such a configuration thatwith respect to the high frequency separation unit 112 shown in FIG. 5of the above-mentioned first embodiment, a first smoothing unit 1300constituting noise reducing means and first smoothing means and a secondsmoothing unit 1301 constituting noise reducing means and secondsmoothing means are added. Other basic configuration is similar to thatof the high frequency separation unit 112 shown in FIG. 5. Therefore,the same components are allocated with the same names and referencenumerals to appropriately omit the description thereof, and only adifferent part will be mainly described.

The determination unit 409 is connected to the first smoothing unit 1300and the second smoothing unit 1301. The first smoothing unit 1300 isconnected to the edge emphasis unit 1202. The second smoothing unit 1301is connected to the buffer 114.

The control unit 118 is bi-directionally connected to the firstsmoothing unit 1300 and the second smoothing unit 1301 to control theseunits.

Subsequently, a description will be given of the action of the highfrequency separation unit 1201.

The low frequency component extraction unit 400 sequentially extractsthe low frequency components from the buffer 110 on the basis of thecontrol of the control unit 118. It should be noted that according tothe present embodiment, as described above, it is supposed to use thediscrete cosine transform of the 64×64 pixels. Then, the low frequencycomponent extraction unit 400 extracts frequency components equal to orsmaller than a predetermined n-th order among the frequency componentsat the respective orders shown in FIG. 27B as the low frequencycomponents from the respective regions of the 64×64 pixels.

Regarding the extracted low frequency components, the noise amount iscalculated via the parameter selection unit 404 and the interpolationunit 405 similarly to the above-mentioned first embodiment. Then, theinterpolation unit 405 transfers the calculated noise amount to theupper limit and lower limit setting unit 408.

The high frequency component extraction unit 406 extracts frequencycomponents at equal to or larger than the (n+1)-th order from therespective regions of the 64×64 pixels corresponding to the lowfrequency components extracted by the low frequency component extractionunit 400 as the high frequency components on the basis of the control ofthe control unit 118.

The average calculation unit 407 separates the high frequency componentsfor each order to calculate the respective average values AV on thebasis of the control of the control unit 118 and transfers thecalculated average value AV to the upper limit and lower limit settingunit 408.

On the basis of the control of the control unit 118, by using theaverage value AV from the average calculation unit 407 and the noiseamount N from the interpolation unit 405, the upper limit and lowerlimit setting unit 408 sets an upper limit App_Up and a lower limitApp_Low for distinguishing the valid component and the invalid componentas represented by Numeric Expression 3 as follows for each order.

The upper limit and lower limit setting unit 408 transfers the thus setupper limit App_Up and the lower limit App_Low to the determination unit409, transfers the average value AV to the second smoothing unit 1301,and transfers the average value AV and the noise amount N to the firstsmoothing unit 1300, respectively.

On the basis of the control of the control unit 118, the determinationunit 409 reads the high frequency components from the high frequencycomponent extraction unit 406, and also reads the upper limit App_Up andthe lower limit App_Low corresponding to the order of the high frequencycomponents from the upper limit and lower limit setting unit 408. Then,in a case where the high frequency component exceeds the upper limitApp_Up or falls short of the lower limit App_Low, the determination unit409 determines that the high frequency component is the valid componentand transfers the high frequency components to the first smoothing unit1300.

On the other hand, in a case where the high frequency component is inthe range between the upper limit App_Up and the lower limit App_Low,the determination unit 409 determines that the high frequency componentis the invalid component caused by the noise and transfers the highfrequency component to the second smoothing unit 1301.

The second smoothing unit 1301 performs a processing of substituting thehigh frequency component (herein, the high frequency component is set asP) with the average value AV from the upper limit and lower limitsetting unit 408 as shown in Numeric Expression 7 below.P=AV  [Expression 7]

Also, the first smoothing unit 1300 uses the average value AV from theupper limit and lower limit setting unit 408 and the noise amount N toperform the correction on the high frequency component P. The correctionhas two types of processings. First, in a case where the high frequencycomponent exceeds the upper limit App_Up, the first smoothing unit 1300performs a correction as shown in Numeric Expression 8 below.P=AV−N/2  [Expression 8]

On the other hand, in a case where the high frequency components fallsshort of the lower limit App_Low, the first smoothing unit 1300 performsa correction as shown in Numeric Expression 9 below.P=AV+N/2  [Expression 9]

Then, the processing result obtained by the first smoothing unit 1300 istransferred to the edge emphasis unit 1202, and the processing resultobtained by the second smoothing unit 1301 is transferred to the buffer114, respectively.

Therefore, only the high frequency component determined as the validcomponent is transferred via the edge emphasis unit 1202 to thegradation processing unit 113, and the gradation processing isperformed. On the other hand, the high frequency component determined asthe invalid component is transferred to the buffer 114 withoutperforming the gradation processing thereon.

It should be noted that according to the present embodiment too,similarly to the above-mentioned first and second embodiments, the imageprocessing system in which the image pickup unit is separately providedmay be used.

Also, in the above, it is supposed to perform the processing by way ofthe hardware, but the configuration is not necessarily limited to theabove. For example, the image signal from the CCD 112 is recorded in therecording medium such as a memory card as raw data without applying theprocess, and also the associated information such as image pickupconditions (for example, the temperature of the image pickup device, theexposure conditions, and the like, for each shooting operation from thecontrol unit 118) is recorded in the recording medium as the headerinformation. Then, the processing can be performed as the computer isallowed to execute the image processing program which is separatesoftware to instruct the computer to read the information of therecording medium. It should be noted that the transmission of variouspieces of information from the image pickup unit to the computer is notnecessarily performed via the recording medium and may be performed viaa communication line or the like.

FIG. 29 is a flow chart showing a main routine of an image processingprogram.

It should be noted that in FIG. 29, processing steps basicallysubstantially identified with the processing shown in FIG. 11 of theabove-mentioned first embodiment are allocated with the same stepnumbers.

When the processing is started, first, the image signal is read, andalso the header information such as the temperature and the exposureconditions of the image pickup device is read (step S1).

Next, by performing the frequency decomposition such as the discretecosine transform, the high frequency component and the low frequencycomponent are obtained (step S2).

Subsequently, as shown in FIG. 12, the conversion characteristic iscalculated (step S3).

Furthermore, as is described below with reference to FIG. 30, the highfrequency component is separated into the invalid component caused bythe noise and the other valid component (step S80).

Then, as shown in FIG. 14, the gradation processing is performed on thelow frequency component and the valid component in the high frequencycomponent (step S5).

Next, on the basis of the low frequency component on which the gradationprocessing has been performed, the valid component in the high frequencycomponent on which the gradation processing has been performed, and theinvalid component in the high frequency component, the image signal onwhich the gradation conversion has been performed is synthesized (stepS6).

Subsequently, the signal processing such as a known compressionprocessing is performed (step S7).

Then, the image signal after the processing is output (step S8), and theprocessing is ended.

FIG. 30 is a flow chart showing the processing for the high frequencyseparation in the above-mentioned step S80.

It should be noted that in FIG. 30, processing steps basicallysubstantially identified with the processing shown in FIG. 13 of theabove-mentioned first embodiment are allocated with the same stepnumbers.

When the processing is started, first, the low frequency components aresequentially extracted for each pixel (step S20).

Next, from the read header information, the information such as thetemperature and the gain of the image pickup device is set. At thistime, if a necessary parameter does not exist for the headerinformation, a pre-set standard value is assigned to the relevantinformation (step S21).

Subsequently, the parameter related to the reference noise model is read(step S22).

Then, on the basis of the parameter of the reference noise model, thenoise amount related to the low frequency component is calculatedthrough the interpolation processing (step S23).

After that, as illustrated in FIG. 27B, the high frequency componentscorresponding to the low frequency components are sequentially extracted(step S24).

Next, the average values of the high frequency components correspondingto the low frequency components are calculated for each order (stepS25).

Subsequently, on the basis of the average value and the noise amount,the upper limit and the lower limit are set as shown in NumericExpression 3 (step S26).

Then, in a case where the high frequency component is in the rangebetween the upper limit and the lower limit, it is determined that thehigh frequency component is the invalid component caused by the noise,and in a case where the high frequency component exceeds the upper limitor falls short of the lower limit, it is determined that the highfrequency component is the valid component (step S90).

At this time, in a case where it is determined that the high frequencycomponent is the valid component, the correction processing shown inNumeric Expression 8 or Numeric Expression 9 is performed on the highfrequency component (step S91).

On the other hand, in step S90, in a case where it is determined thatthe high frequency component is the invalid component, the correctionprocessing shown in Numeric Expression 7 is performed on the highfrequency component (step S92).

When the processing in step S91 or S92 is ended, the valid component andthe invalid component are output while being separated from each other(step S93).

Then, it is determined whether or not the processing for all the highfrequency components has been completed (step S29). In a case where itis determined that the processing has not been completed, the flow isreturned to the above-mentioned step S24 to repeat the above-mentionedprocessing.

On the other hand, in the step S29, in a case where it is determinedthat the processing for all the high frequency components has beencompleted, it is determined whether or not the processing for all thelow frequency components has been completed (step S30). In a case whereit is determined that the processing has not been completed, the flow isreturned to the above-mentioned step S20 to repeat the above-mentionedprocessing. On the other hand, in a case where it is determined that theprocessing has been completed, the flow is returned to the processingshown in FIG. 29.

It should be noted that in the above description, the configuration ofusing the discrete cosine transform for the frequency decomposition andthe frequency synthesis is adopted, but the configuration is notnecessarily limited to the above. For example, similarly to theabove-mentioned first embodiment, a configuration of using the wavelettransform can be adopted, and similarly to the second embodimentdescribed above, a configuration of using the low-pass filter and thedifference filter in combination can also be adopted.

Furthermore, in the above description, the configuration of processingthe monochrome image signal is adopted, but the configuration is notnecessarily limited to the above. For example, similarly to the secondembodiment described above, a configuration of calculating the luminancesignals from the color image signal obtained from the color image pickupdevice for the processing can also be adopted.

According to the third embodiment described above, only the highfrequency component where the influence of the noise prominentlyvisually appears is separated into the invalid component and the validcomponent. The gradation processing is performed on the valid component,and the gradation processing is not performed on the invalid component,and an increase in noise accompanying with the gradation processing issuppressed. Thus, it is possible to generate the high quality imagesignal.

Also, as the low frequency component is excluded from the target of theprocessing after being separated into the valid component and theinvalid component, the possibility of generating the adverse effectaccompanying with the processing is decreased, and it is possible toimprove the stability.

Furthermore, as the image signal is synthesized with the invalidcomponent, it is possible to obtain the image signal with little senseof visual discomfort, and the stability and reliability of theprocessing can be improved.

Also, the discrete cosine transform is excellent at the separation ofthe frequency, and it is therefore possible to perform the high accuracyprocessing.

As the gradation conversion curve is adaptively and also independentlycalculated for each region from the low frequency component of the imagesignal, it is possible to perform the gradation conversion at the highaccuracy on various image signals.

Then, as the gradation conversion is performed on the high frequencycomponent on which the noise reducing processing has been performed, anincrease in noise accompanying with the gradation processing issuppressed. Thus, it is possible to generate the high quality imagesignal.

Also, as the correction processing is performed on the valid componentin the high frequency component and the smoothing processing isperformed on the invalid component in the high frequency component, thegeneration of the discontinuity accompanying with the noise reducingprocessing is prevented, and it is possible to generate the high qualityimage signal.

Furthermore, as the edge emphasis processing is performed only on thevalid component in the high frequency component and the edge emphasisprocessing is not performed on the invalid component in the highfrequency component, it is possible to emphasize only the edge componentwithout emphasizing the noise component. With the configuration, it ispossible to generate the high quality image signal.

Fourth Embodiment

FIGS. 31 to 36 illustrate a fourth embodiment of the present invention,and FIG. 31 is a block diagram of a configuration of an image processingsystem.

According to the fourth embodiment, the same configuration as that ofthe above-mentioned first to third embodiments is allocated with thesame reference numerals to appropriately omit the description thereof,and only a different part will be mainly described.

The image processing system according to the present embodiment has sucha configuration that with respect to the above-mentioned imageprocessing system illustrated in FIG. 1 according to the firstembodiment, a noise reducing unit 1400 constituting separation means andnoise reducing means, a difference unit 1401 constituting separationmeans and difference means, and a signal synthesis unit 1403constituting synthesis means and signal synthesis means are added, thegradation processing unit 113 is replaced by a gradation processing unit1402 constituting conversion means and gradation processing means, andthe frequency decomposition unit 109, the high frequency separation unit112, and the frequency synthesis unit 115 are omitted. Other basicconfiguration is similar to that of the above-mentioned firstembodiment. Therefore, the same components are allocated with the samenames and reference numerals to appropriately omit the descriptionthereof, and only a different part will be mainly described.

The buffer 105 is connected to the exposure control unit 106, the focuscontrol unit 107, the noise reducing unit 1400, and the difference unit1401.

The noise reducing unit 1400 is connected to the buffer 110. The buffer110 is connected to the conversion characteristic calculation unit 111,the difference unit 1401, and the gradation processing unit 1402.

The conversion characteristic calculation unit 111 is connected to thegradation processing unit 1402. The difference unit 1401 is connected tothe buffer 114. The gradation processing unit 1402 is connected to thebuffer 114. The buffer 114 is connected via the signal synthesis unit1403 to the signal processing unit 116.

The control unit 118 is also bi-directionally connected to the noisereducing unit 1400, the difference unit 1401, the gradation processingunit 1402, and the signal synthesis unit 1403 to control these units.

Next, the action of the image processing system illustrated in FIG. 31is basically similar to that of the first embodiment, and therefore onlya different part will be mainly described along the flow of the imagesignal.

The image signal in the buffer 105 is transferred to the noise reducingunit 1400.

The noise reducing unit 1400 performs the noise reducing processing onthe basis of the control of the control unit 118 and transfers the imagesignal after the noise reducing processing as the valid component to thebuffer 110.

The conversion characteristic calculation unit 111 reads the validcomponent from the buffer 110, and similarly to the above-mentionedfirst embodiment, calculates the gradation characteristic used for thegradation conversion processing. It should be noted that according tothe present embodiment, for the gradation conversion processing, forexample, it is supposed to use a space-variant processing using aplurality of gradation characteristics different for each region of a64×64 pixel unit. Then, the conversion characteristic calculation unit111 transfers the calculated the gradation characteristic to thegradation processing unit 1402.

On the basis of the control of the control unit 118, the difference unit1401 reads the image signal before the noise reducing processing fromthe buffer 105, and also reads the image signal after the noise reducingprocessing from the buffer 110 as the valid component to perform aprocessing of taking a difference thereof. The difference unit 1401transfers a signal obtained as the result of taking the difference asthe invalid component to the buffer 114.

The gradation processing unit 1402 reads the valid component from thebuffer 110 and the gradation characteristic from the conversioncharacteristic calculation unit 111, respectively, on the basis of thecontrol of the control unit 118. Then, on the basis of theabove-mentioned gradation characteristic, the gradation processing unit1402 performs the gradation processing on the above-mentioned validcomponent. The gradation processing unit 1402 transfers the validcomponent on which the gradation processing has been performed to thebuffer 114.

The signal synthesis unit 1403 reads the valid component on which thegradation processing has been performed and the invalid component fromthe buffer 114 on the basis of the control of the control unit 118 andadds these components, so that the image signal on which the gradationconversion has been performed is synthesized. The signal synthesis unit1403 transfers the image signal thus synthesized to the signalprocessing unit 116.

The signal processing unit 116 performs a known compression processingor the like on the image signal from the signal synthesis unit 1403 andtransfers the signal after the processing to the output unit 117 on thebasis of the control of the control unit 118.

The output unit 117 records and saves the image signal output from thesignal processing unit 116 in the recording medium such as a memorycard.

Next, FIG. 32 is a block diagram of a configuration example of the noisereducing unit 1400.

The noise reducing unit 1400 is configured by including an image signalextraction unit 1500, an average calculation unit 1501 constitutingnoise estimation means and average calculation means, a gain calculationunit 1502 constituting noise estimation means and collection means, astandard value assigning unit 1503 constituting noise estimation meansand assigning means, a noise LUT 1504 constituting noise estimationmeans and table conversion means, an upper limit and lower limit settingunit 1505 constituting setting means and upper limit and lower limitsetting means, a determination unit 1506 constituting determinationmeans, a first smoothing unit 1507 constituting first smoothing means,and a second smoothing unit 1508 constituting second smoothing means.

The buffer 105 is connected to the image signal extraction unit 1500.The image signal extraction unit 1500 is connected to the averagecalculation unit 1501 and the determination unit 1506.

The average calculation unit 1501, the gain calculation unit 1502, andthe standard value assigning unit 1503 are connected to the noise LUT1504. The noise LUT 1504 is connected to the upper limit and lower limitsetting unit 1505. The upper limit and lower limit setting unit 1505 isconnected to the determination unit 1506, the first smoothing unit 1507,and the second smoothing unit 1508.

The determination unit 1506 is connected to the first smoothing unit1507 and the second smoothing unit 1508. The first smoothing unit 1507and the second smoothing unit 1508 are connected to the buffer 110.

The control unit 118 is bi-directionally connected to the image signalextraction unit 1500, the average calculation unit 1501, the gaincalculation unit 1502, the standard value assigning unit 1503, the noiseLUT 1504, the upper limit and lower limit setting unit 1505, thedetermination unit 1506, the first smoothing unit 1507, and the secondsmoothing unit 1508 to control these units.

Subsequently, a description will be given of the action of the noisereducing unit 1400.

The image signal extraction unit 1500 sequentially extracts the targetpixel on which the noise reducing processing should be performed andneighboring pixels of, for example, 3×3 pixels including the targetpixel from the buffer 105 on the basis of the control of the controlunit 118. The image signal extraction unit 1500 transfers the targetpixel and the neighboring pixels to the average calculation unit 1501,and the target pixel to the determination unit 1506, respectively.

The average calculation unit 1501 reads the target pixel and theneighboring pixels from the image signal extraction unit 1500 andcalculates the average value AV thereof on the basis of the control ofthe control unit 118. The average calculation unit 1501 transfers thecalculated average value AV to the noise LUT 1504.

The gain calculation unit 1502 calculates the gain information in theamplification unit 103 to be transferred to the noise LUT 1504 on thebasis of the information related to the ISO sensitivity and the exposurecondition transferred from the control unit 118.

Also, the control unit 118 obtains temperature information of the CCD102 from the temperature sensor 120 and transferred the thus obtainedtemperature information to the noise LUT 1504.

On the basis of the control of the control unit 118, in a case where atleast one of the above-mentioned gain information and the temperatureinformation cannot be obtained, the standard value assigning unit 1503transfers a standard value of the information that cannot be obtained tothe noise LUT 1504.

The noise LUT 1504 is a look up table where a relation among the signalvalue level of the image signal, the gain of the image signal, theoperation temperature of the image pickup device, and the noise amountis recorded. The look up table is designed, for example, by using thetechnology disclosed in Japanese Unexamined Patent ApplicationPublication No. 2004-128985.

The noise LUT 1504 outputs the noise amount N on the basis of theaverage value AV related to the target pixel from the averagecalculation unit 1501, the gain information from the gain calculationunit 1502 or the standard value assigning unit 1503, and the temperatureinformation from the control unit 118 or the standard value assigningunit 1503. The noise amount N and the average value AV from the averagecalculation unit 1501 are transferred from the noise LUT 1504 to theupper limit and lower limit setting unit 1505.

On the basis of the control of the control unit 118, the upper limit andlower limit setting unit 1505 uses the average value AV and the noiseamount N from the noise LUT 1504 to set the upper limit App_Up and thelower limit App_Low for identifying whether the target pixel belongs tothe noise or not as shown in Numeric Expression 3.

The upper limit and lower limit setting unit 1505 transfers the thus setupper limit App_Up and the lower limit App_Low to the determination unit1506, transfers the average value AV to the second smoothing unit 1508,and transfers the average value AV and the noise amount N to the firstsmoothing unit 1507, respectively.

The determination unit 1506 reads the target pixel from the image signalextraction unit 1500 and the upper limit App_Up and the lower limitApp_Low from the upper limit and lower limit setting unit 1505,respectively, on the basis of the control of the control unit 118. Then,in a case where the target pixel exceeds the upper limit App_Up or fallsshort of the lower limit App_Low, the determination unit 1506 determinesthat the target pixel does not belong to the noise and transfers thetarget pixel to the first smoothing unit 1507.

On the other hand, in a case where the target pixel is in the rangebetween the upper limit App_Up and the lower limit App_Low, thedetermination unit 1506 determines that the target pixel belongs to thenoise and transfers the target pixel to the second smoothing unit 1508.

The second smoothing unit 1508 performs the processing of substitutingthe target pixel (herein, the target pixel is set as P) with the averagevalue AV from the upper limit and lower limit setting unit 1505 as shownin Numeric Expression 7.

Also, the first smoothing unit 1507 uses the average value AV and thenoise amount N from the upper limit and lower limit setting unit 1505 toperform the correction on the target pixel P. The correction has twotypes of processings. In a case where the target pixel P exceeds theupper limit App_Up, the first smoothing unit 1507 performs thecorrection shown in Numeric Expression 8. On the other hand, the firstsmoothing unit 1507 performs the correction shown in Numeric Expression9 in a case where the target pixel P falls short of the lower limitApp_Low.

Then, the processing result obtained by the first smoothing unit 1507and the processing result obtained by the second smoothing unit 1508 areboth transferred to the buffer 110.

Next, FIG. 33 is a block diagram of a configuration example of thegradation processing unit 1402.

The gradation processing unit 1402 has such a configuration that withrespect to the gradation processing unit 113 shown in FIG. 6 of theabove-mentioned first embodiment, the low frequency component extractionunit 500, the high frequency component extraction unit 504 is omitted,and an image signal extraction unit 1600 constituting extraction meansis added. Other basic configuration is similar to that of the gradationprocessing unit 113 shown in FIG. 6. Therefore, the same components areallocated with the same names and reference numerals to appropriatelyomit the description thereof, and only a different part will be mainlydescribed.

The buffer 110 is connected to the image signal extraction unit 1600.The image signal extraction unit 1600 is connected to the distancecalculation unit 501 and the gradation conversion unit 505.

The control unit 118 is also bi-directionally connected to the imagesignal extraction unit 1600 to control the unit.

Subsequently, a description will be given of the action of the gradationprocessing unit 1402.

The image signal extraction unit 1600 sequentially extracts the imagesignals after the noise reducing processing as valid components from thebuffer 110 for each pixel on the basis of the control of the controlunit 118. The image signal extraction unit 1600 transfers the extractedvalid component to the distance calculation unit 501 and the gradationconversion unit 505.

After that, similarly to the above-mentioned first embodiment, thedistance calculation unit 501 and the gradation conversion equationsetting unit 502 sets the gradation conversion equation with respect tothe target pixel as shown in Numeric Expression 4. Then, the gradationconversion equation setting unit 502 transfers the set gradationconversion equation to the buffer 503.

On the basis of the control of the control unit 118, the gradationconversion unit 505 reads the valid component from the image signalextraction unit 1600 and also reads the gradation conversion equationfrom the buffer 503 to perform the gradation conversion on the validcomponent. The gradation conversion unit 505 transfers the validcomponent after the gradation conversion to the buffer 114.

It should be noted that according to the present embodiment too,similarly to the above-mentioned first to third embodiments, the imageprocessing system in which the image pickup unit is separately providedmay be used.

Also, in the above, it is supposed to perform the processing by way ofthe hardware, but the configuration is not necessarily limited to theabove. For example, the image signal from the CCD 112 is recorded in therecording medium such as a memory card as raw data without applying theprocess, and also the associated information such as image pickupconditions (for example, the temperature of the image pickup device, theexposure conditions, and the like, for each shooting operation from thecontrol unit 118) is recorded in the recording medium as the headerinformation. Then, the processing can be performed as the computer isallowed to execute the image processing program which is separatesoftware to instruct the computer to read the information of therecording medium. It should be noted that the transmission of variouspieces of information from the image pickup unit to the computer is notnecessarily performed via the recording medium and may be performed viaa communication line or the like.

FIG. 34 is a flow chart showing a main routine of an image processingprogram.

It should be noted that in FIG. 34, processing steps basicallysubstantially identified with the processing shown in FIG. 11 of theabove-mentioned first embodiment are allocated with the same stepnumbers.

When the processing is started, first, the image signal is read, andalso the header information such as the temperature and the exposureconditions of the image pickup device is read (step S1).

Next, as is described below with reference to FIG. 35, the noisereducing processing is performed to calculate the image signal after thenoise reducing processing as the valid component (step S100).

Subsequently, as shown in FIG. 12, the conversion characteristic iscalculated (step S3).

Furthermore, from the difference between the image signal and the imagesignal after the noise reducing processing, the invalid component iscalculated (step S101).

Then, as is described below with reference to FIG. 36, the gradationprocessing is performed on the valid component (step S102).

Next, on the basis of the valid component on which the gradationprocessing has been performed and the invalid component, the imagesignal on which the gradation conversion has been performed issynthesized (step S103).

Subsequently, the signal processing such as a known compressionprocessing is performed (step S7).

Then, the image signal after the processing is output (step S8), and theprocessing is ended.

FIG. 35 is a flow chart showing the processing for the noise reductionin the above-mentioned step S100.

When the processing is started, first, the target pixel on which thenoise reducing processing should be performed and neighboring pixels,for example, of 3×3 pixels including the target pixel are sequentiallyextracted (step S110).

Next, an average value of the target pixel and the neighboring pixels iscalculated (step S111).

Subsequently, from the read header information, the information such asthe temperature and the gain of the image pickup device is set. At thistime, if a necessary parameter does not exist for the headerinformation, a pre-set standard value is assigned to the relevantinformation (step S112).

Then, the table related to the noise amount where a relation among thesignal value level of the image signal, the gain of the image signal,the operation temperature of the image pickup device, and the noiseamount is recorded is read (step S113).

Furthermore, on the basis of the table related to the noise amount, thenoise amount is calculated (step S114).

After that, on the basis of the average value and the noise amount, theupper limit and the lower limit are set as shown in Numeric Expression 3(step S115).

Next, it is determined whether the target pixel belongs to the noise ornot through the comparison with the upper limit and the lower limit(step S116).

At this time, in a case where the target pixel exceeds the upper limitor falls short of the lower limit, it is determined that the targetpixel does not belong to the noise, and the correction processing shownin Numeric Expression 8 or Numeric Expression 9 is performed on thetarget pixel (step S117).

On the other hand, in step S116, in a case where the target pixel is inthe range between the upper limit and the lower limit, it is determinedthat the target pixel belongs to the noise, the correction processingshown in Numeric Expression 7 is performed on the target pixel (stepS118).

Then, the corrected target pixel is output as the pixel after the noisereducing processing (step S119).

After that, the image signal after the noise reducing processing is setas the valid component, and it is determined whether the processing hasbeen completed for all the valid components or not (step S120). In acase where it is determined that the processing has not been completed,the flow is returned to the above-mentioned step S110 to repeat theabove-mentioned processing. On the other hand, in a case where it isdetermined that the processing has been completed, the flow is returnedto the processing shown in FIG. 34.

FIG. 36 is a flow chart showing the gradation processing in theabove-mentioned step S102.

It should be noted that in FIG. 36, processing steps basicallysubstantially identified with the processing shown in FIG. 14 of theabove-mentioned first embodiment are allocated with the same stepnumbers.

When the processing is started, first, the image signals after the noisereducing processing are sequentially extracted as valid components foreach pixel (step S130).

Next, as illustrated in FIG. 8, the distances between the target pixelof the valid component and the centers of the four neighboring regionsare calculated (step S41).

Subsequently, the gradation conversion curves in the four neighboringregions are read (step S42).

Furthermore, as shown in Numeric Expression 4, the gradation conversionequation with respect to the target pixel is set (step S43).

Then, by applying the gradation conversion equation shown in NumericExpression 4 with respect to the target pixel of the valid component,the gradation conversion is performed (step S47).

Next, the target pixel on which the gradation processing has beenperformed is output (step S48).

After that, it is determined whether the processing has been completedfor all the image signals after the noise reducing processing or not(step S131). In a case where it is determined that the processing hasnot been completed, the flow is returned to the above-mentioned stepS130 to repeat the above-mentioned processing. On the other hand, in acase where it is determined that the processing has been completed, theflow is returned to the processing shown in FIG. 34.

It should be noted that in the above description, the configuration ofprocessing the monochrome image signal is adopted, but the configurationis not necessarily limited to the above. For example, similarly to thesecond embodiment described above, it is possible to adopt aconfiguration of processing the color image signal obtained from thecolor image pickup device.

According to the fourth embodiment described above, the gradationprocessing is performed only on the image signal after the noisereduction, and an increase in noise accompanying with the gradationprocessing is suppressed. Thus, it is possible to generate the highquality image signal.

Also, as the conversion characteristic is calculated on the basis of theimage signal after the noise reduction, the appropriate conversioncharacteristic with little influence from the noise can be calculated,and it is possible to improve the stability and reliability of theprocessing. At this time, as the gradation conversion curve isadaptively calculated from the image signal after the noise reduction,it is possible to perform the high accuracy gradation conversion onvarious types of image signals.

Furthermore, the present embodiment corresponds to the processing systemin which the gradation conversion processing is combined with the noisereducing processing. Therefore, the affinity and compatibility with theexisting system are high, and the present embodiment can be applied to alarge number of image processing systems. Furthermore, the higherperformance can be achieved as a whole, and the system scale can bereduced, which leads to the realization of the lower cost.

Then, the image signal after the noise reduction on which the gradationprocessing has been performed and the invalid component are synthesizedwith each other. Thus, the error generated in the noise reducingprocessing can be suppressed, and it is possible to perform the stablegradation processing. Also, it is possible to generate the high qualityimage signal with little sense of visual discomfort.

In addition, as the gradation conversion curve is adaptively obtained,it is possible to perform the high accuracy gradation conversion onvarious types of image signals.

Also, as the gradation conversion curve is obtained independently foreach region, the degree of freedom is further improved, and also it ispossible to obtain the high quality image signals for scenes with alarge contrast.

It should be noted that the present invention is not limited to theembodiments described above, and various modifications and applicationscan of course be made without departing from the gist of the presentinvention.

1. An image processing system arranged to perform a gradation conversionon an image signal, the image processing system comprising: a frequencydecomposition unit for decomposing the image signal into a highfrequency component and a low frequency component; a high frequencyseparation unit for separating the high frequency component into aninvalid component caused by noise and other valid component; aconversion characteristic calculation unit for calculating a conversioncharacteristic used for the gradation conversion on the basis of the lowfrequency component; a gradation processing unit for respectivelyperforming gradation processing on the low frequency component andgradation processing on the valid component in the high frequencycomponent by using the conversion characteristic, and respectivelyoutputting the low frequency component on which the gradation processinghas been performed and the valid component in the high frequencycomponent on which the gradation processing has been performed; and afrequency synthesis unit for synthesizing the invalid component in thehigh frequency component, the low frequency component on which thegradation processing has been performed and the valid component in thehigh frequency component on which the gradation processing has beenperformed.
 2. The image processing system according to claim 1, furthercomprising a Y/C separation unit configured to separate, in a case wherethe image signal is a color image signal, the color image signal into aluminance signal and a color signal, wherein the image signal dealt withby the frequency decomposition unit, the high frequency separation unit,the conversion characteristic calculation unit, the gradation processingunit, and the frequency synthesis unit is the luminance signal separatedby the Y/C separation unit.
 3. The image processing system according toclaim 2, further comprising a Y/C synthesis unit configured tosynthesize the color signal on which the gradation conversion has beenperformed on the basis of the luminance signal on which the gradationconversion has been performed and the color signal.
 4. The imageprocessing system according to claim 2, wherein the color image signalis a color image signal obtained from one of a single image pickupdevice in which an R (red), G (green), and B (blue) Bayer-type primarycolor filter is arranged on a front face and a single image pickupdevice in which a Cy (cyan), Mg (magenta), Ye (yellow), and G (green)color-difference line-sequential type complementary color filter isarranged on a front face.
 5. The image processing system according toclaim 1, wherein the frequency decomposition unit uses one of a wavelettransform, a Fourier transform, and a discrete cosine transform todecompose the image signal into the high frequency component and the lowfrequency component.
 6. The image processing system according to claim1, wherein the frequency decomposition unit uses a low-pass filter and adifference filter to decompose the image signal into the high frequencycomponent and the low frequency component.
 7. The image processingsystem according to claim 1, wherein the frequency decomposition unituses a Gaussian filter and a Laplacian filter to decompose the imagesignal into the high frequency component and the low frequencycomponent.
 8. The image processing system according to claim 1, whereinthe conversion characteristic calculation unit includes: division unitconfigured to divide the low frequency component into a plurality ofregions; correct range extraction unit configured to extract a correctexposure range on the basis of a signal value of the low frequencycomponent for each region; edge calculation unit configured to calculatean edge amount regarding the correct exposure range for each region;histogram creation unit configured to create a histogram on the basis ofthe edge amount for each region; and gradation conversion curvecalculation unit configured to calculate a gradation conversion curve asthe conversion characteristic on the basis of the histogram for eachregion.
 9. The image processing system according to claim 8, wherein thegradation processing unit includes: first extraction unit configured tosequentially extract low frequency component target pixels which aretargets of the gradation processing from the low frequency component;second extraction unit configured to sequentially extract validcomponent target pixels corresponding to positions of the low frequencycomponent target pixels from the valid component in the high frequencycomponent; distance calculation unit configured to calculate distanceinformation between the low frequency component target pixel and acenter of the region located in a neighborhood of the low frequencycomponent target pixel; gradation conversion equation setting unitconfigured to set a gradation conversion equation used for the gradationconversion on the basis of the gradation conversion curve of apredetermined number of regions located in a neighborhood of the lowfrequency component target pixel and the distance information; andgradation conversion unit configured to perform a gradation conversionon the low frequency component target pixel and the valid componenttarget pixel on the basis of the set gradation conversion equation. 10.The image processing system according to claim 9, wherein the gradationprocessing unit further includes control unit configured to perform, ina case where the valid component target pixel corresponding to theposition of the low frequency component target pixel does not exist, acontrol to cancel the gradation conversion on the valid component targetpixel.
 11. The image processing system according to claim 1, furthercomprising noise reducing unit configured to perform a noise reducingprocessing on the high frequency component.
 12. The image processingsystem according to claim 1, further comprising edge emphasis unitconfigured to perform an edge emphasis processing on the valid componentin the high frequency component.
 13. An image processing system arrangedto perform a gradation conversion on an image signal, the imageprocessing system comprising: a separation unit adapted to separate theimage signal into an invalid component caused by noise and other validcomponent; a conversion unit adapted to perform the gradation conversionon the valid component; and a synthesis unit adapted to synthesize animage signal on which the gradation conversion has been performed on thebasis of the valid component on which the gradation conversion has beenperformed and the invalid component, the separation unit comprising: afrequency decomposition unit adapted to decompose the image signal intoa high frequency component and a low frequency component; and a highfrequency separation unit adapted to separate the high frequencycomponent into an invalid component caused by noise and other validcomponent, and the conversion unit comprising: a conversioncharacteristic calculation unit adapted to calculate a conversioncharacteristic used for the gradation conversion on the basis of the lowfrequency component; and a gradation processing unit adapted to performa gradation processing on the low frequency component and the validcomponent in the high frequency component by using the conversioncharacteristic, wherein the conversion characteristic calculation unitincludes: a correct range extraction unit configured to extract acorrect exposure range on the basis of a signal value of the lowfrequency component; an edge calculation unit configured to calculate anedge amount regarding the correct exposure range; a histogram creationunit configured to create a histogram on the basis of the edge amount;and a gradation conversion curve calculation unit configured tocalculate a gradation conversion curve as the conversion characteristicon the basis of the histogram.
 14. The image processing system accordingto claim 13, wherein the conversion characteristic calculation unitfurther includes: a region-of-interest setting unit configured to set aregion-of-interest from the low frequency component; a weighting factorsetting unit configured to set a weighting factor regarding theregion-of-interest; and a histogram correction unit configured tocorrect the histogram on the basis of the weighting factor, wherein thegradation conversion curve calculation unit calculates the gradationconversion curve on the basis of the corrected histogram.
 15. The imageprocessing system according to claim 13, wherein the gradationprocessing unit includes: a low frequency component extraction unit tosequentially extract low frequency component target pixels which aretargets of the gradation processing from the low frequency component, ahigh frequency component extraction unit configured to sequentiallyextract valid component target pixels corresponding to positions of thelow frequency component target pixels from the valid component in thehigh frequency component; and a gradation conversion unit configured toperform a gradation conversion on the low frequency component targetpixels and the valid component target pixels on the basis of thegradation conversion curve.
 16. The image processing system according toclaim 15, further comprising a control unit configured to perform, in acase where the valid component target pixel corresponding to theposition of the low frequency component target pixel does not exist, acontrol to cancel the gradation conversion on the valid component targetpixel.
 17. An image processing system arranged to perform a gradationconversion on an image signal, the image processing system comprising: aseparation unit adapted to separate the image signal into an invalidcomponent caused by noise and other valid component; a conversion unitadapted to perform the gradation conversion on the valid component; anda synthesis unit adapted to synthesize an image signal on which thegradation conversion has been performed on the basis of the validcomponent on which the gradation conversion has been performed and theinvalid component, the separation unit comprising: a frequencydecomposition unit adapted to decompose the image signal into a highfrequency component and a low frequency component; and a high frequencyseparation unit adapted to separate the high frequency component into aninvalid component caused by noise and other valid component, and theconversion unit comprising: a conversion characteristic calculation unitadapted to calculate a conversion characteristic used for the gradationconversion on the basis of the low frequency component; and a gradationprocessing unit adapted to perform a gradation processing on the lowfrequency component and the valid component in the high frequencycomponent by using the conversion characteristic, wherein the highfrequency separation unit includes: a noise estimation unit adapted toestimate a noise amount of the high frequency component on the basis ofthe low frequency component; a setting unit adapted to set a permissiblerange on the basis of the noise amount and the high frequency component;and a determination unit configured to determine whether the highfrequency component belongs to the invalid component or the validcomponent on the basis of the permissible range.
 18. The imageprocessing system according to claim 17, wherein the high frequencyseparation unit includes: a gain calculation unit configured to collectinformation related to a temperature value of an image pickup deviceused when the image signal is picked up and a gain value with respect tothe image signal; a standard value assignment unit configured to assigna standard value with respect to information which cannot be collectedby the collection means; a noise LUT configured to output the noiseamount on the basis of the information from the gain calculation unit orthe standard value assignment unit and the low frequency component orthe image signal; an average calculation unit configured to calculate anaverage value of the high frequency components; an upper limit and lowerlimit setting unit configured to set an upper limit and a lower limitregarding the high frequency component as values representing borders ofa permissible range on the basis of the noise amount and the averagevalue; and a determination unit configured to determine whether the highfrequency component belongs to the invalid component or the validcomponent on the basis of the permissible range.
 19. A non-transitorycomputer-readable storage medium storing a program, when executed by acomputer, causes the computer to perform a gradation conversion on animage signal, comprising the following steps: a frequency decompositionstep of decomposing the image signal into a high frequency component anda low frequency component; a frequency separation step of separating thehigh frequency component into an invalid component caused by noise andother valid component; a conversion characteristic calculation step ofcalculating a conversion characteristic used for the gradationconversion on the basis of the low frequency component; and a gradationprocessing step of respectively performing gradation processing on thelow frequency component and gradation processing on the valid componentin the high frequency component by using the conversion characteristic,and respectively outputting the low frequency component on which thegradation processing has been performed and the valid component in thehigh frequency component on which the gradation processing has beenperformed; and a frequency synthesis step of synthesizing the invalidcomponent in the high frequency component, the low frequency componenton which the gradation processing has been performed and the validcomponent in the high frequency component on which the gradationprocessing has been performed.
 20. The recording medium according toclaim 19, wherein the image processing program instructs the computer tofurther execute a Y/C separation step of separating, in a case where theimage signal is a color image signal, the color image signal into aluminance signal and a color signal, and wherein the image signal dealtwith by the frequency decomposition step, the frequency separation step,the conversion characteristic calculation step, the gradation processingstep, and the frequency synthesis step is the luminance signal separatedin the Y/C separation step.
 21. The recording medium according to claim19, wherein the high frequency separation step includes: a noiseestimation step of estimating a noise amount of the high frequencycomponent on the basis of the low frequency component; a setting step ofsetting a permissible range on the basis of the noise amount and thehigh frequency component; and determination step of determining whetherthe high frequency component belongs to the invalid component or thevalid component on the basis of the permissible range.
 22. The recordingmedium according to claim 19, wherein the image processing programinstructs the computer to further execute a noise reducing step ofperforming a noise reducing processing on the high frequency component.23. The recording medium according to claim 19, wherein the imageprocessing program instructs the computer to further execute an edgeemphasis step of performing an edge emphasis processing on the validcomponent in the high frequency component.
 24. A recording mediumrecording an image processing program for instructing a computer toperform a gradation conversion on an image signal, the image processingprogram instructing the computer to execute: a separation step ofseparating the image signal into an invalid component caused by noiseand other valid component; a conversion step of performing the gradationconversion on the valid component; and a synthesis step of synthesizingan image signal on which the gradation conversion has been performed onthe basis of the valid component on which the gradation conversion hasbeen performed and the invalid component, wherein the separation stepincludes: a frequency decomposition step of decomposing the image signalinto a high frequency component and a low frequency component; and afrequency separation step of separating the high frequency componentinto an invalid component caused by noise and other valid component,wherein the conversion step includes: a conversion characteristiccalculation step of calculating a conversion characteristic used for thegradation conversion on the basis of the low frequency component; and agradation processing step of performing a gradation processing on thelow frequency component and the valid component in the high frequencycomponent by using the conversion characteristic, and wherein theconversion characteristic calculation step includes: a division step ofdividing the low frequency component into a plurality of regions; acorrect range extraction step of extracting a correct exposure range onthe basis of a signal value of the low frequency component for eachregion; an edge calculation step of calculating an edge amount regardingthe correct exposure range for each region; a histogram creation step ofcreating a histogram on the basis of the edge amount for each region;and a gradation conversion curve calculation step of calculating agradation conversion curve as the conversion characteristic on the basisof the histogram for region.
 25. The recording medium according to claim24, wherein the gradation processing step includes: a first extractionstep of sequentially extracting low frequency component target pixelswhich are targets of the gradation processing from the low frequencycomponent; a second extraction step of sequentially extracting validcomponent target pixels corresponding to positions of the low frequencycomponent target pixels from the valid component in the high frequencycomponent; and a distance calculation step of calculating distanceinformation between the low frequency component target pixel and acenter of the region located in a neighborhood of the low frequencycomponent target pixel; a gradation conversion equation setting step ofsetting a gradation conversion equation used for the gradationconversion on the basis of the gradation conversion curve of apredetermined number of regions located in a neighborhood of the lowfrequency component target pixel and the distance information; and agradation conversion step of performing a gradation conversion on thelow frequency component target pixel and the valid component targetpixel on the basis of the set gradation conversion equation.
 26. Therecording medium according to claim 25, wherein the gradation processingstep further includes a control step of performing, in a case where thevalid component target pixel corresponding to the position of the lowfrequency component target pixel does not exist, a control to cancel thegradation conversion on the valid component target pixel.